Yes, I also believe that numbers are infinite and that the universe’s mass is finite. But there are philosophic theories (e.g. Ultrafinitism) that believe numbers aren’t infinite. Ultrafinitism says that number can only exist if it has a physical representation. Therefore you can’t always add one. According to this philosophy, you might not be able add one to the number of elementary particles in the universe, for instance, because you would need one more particle, and you are already using all of them. Nonsense, in my opinion, but that is just my opinion, many disagree.
Ultrafinitsm’s notion is the kind of notion that Kisuke Moto implied here:
Actually, math can trick you in these things. I believe it would be perfectly possible to have an infinite universe, but with small probability to find a “cheese planet” or whatever. The universe could have less matter as you farther from the center in a (non-homogeneous) random way. Up to some point, the probability of finding a large group of particles together (and therefore an entire planet) could approach zero.
I’m saying that as a possibility because I didn’t do the math to find out if that would be the case, it’s just an intuition.