How do we know numbers of past lives and rebirths? I can understand how the Buddha and various arhats and the like can surmise the existence of past lives and rebirth. What I dont see is how they come up with actual numbers of times they have been reborn and the total number of times anyone is and how long the universe has been contracting and expanding and things like that? At some point isn't that delving into the unknowable?
Apparently in Buddhist cosmology, the universe of worlds begins and ends cyclically. That's called an aeon or eon or kalpa. The Visuddhimagga calls this a Maha Kalpa (Great Aeon).

This wikipedia page (without sufficient citations) gives some statistics on the duration of a Maha Kalpa:


  Imagine a huge empty cube at the beginning of a kalpa, approximately
  16 miles in each side. Once every 100 years, you insert a tiny mustard
  seed into the cube. According to the Buddha, the huge cube will be
  filled even before the kalpa ends.
  
  Imagine a gigantic rocky mountain at the beginning of kalpa,
  approximately 16 x 16 x 16 miles (dwarfing Mount Everest). You take a
  small piece of silk and wipe the mountain once every 100 years.
  According to the Buddha, the mountain will be completely depleted even
  before the kalpa ends.


I found that a tiny mustard seed could have a diameter of around 1mm. If I make it easier by assuming that it is a cube with the sides of 1 x 1 x 1mm, then it has 1 cubic mm of volume.

Meanwhile, a huge empty cube of 16 miles on each side has 17 billion cubic mm of volume. So, if this huge empty cube is filled up at 1 cubic mm per 100 years, then how many years does it take to fill up 17 billion cubic mm of volume?

In my calculation, it would be 1.7 trillion years. So, the description is that this huge empty cube will be filled up before the aeon ends. Currently, physicists estimate that the age of our current universe is 13.82 billion years.

The same page also states:


  If you count the total number of sand particles at the depths of the
  Ganges river, from where it begins to where it ends at the sea, even
  that number will be less than the number of passed kalpas.


I'm not sure how many sand particles there are at the depth of the Ganges river from the beginning to the end of it, but I guess there must be at least trillions of particles based on this article. So, according to this description, we have had at least trillions of aeons in the past, with each lasting over 1.7 trillion years each.

In the Assu Sutta, it is taught that the amount of tears that we have shed in past lives exceed the volume of the four great oceans.

According to this article, the average amount of tears shed by a human being in a lifetime is 64 litres. And according to this article, the ocean has 321,003,271 cubic miles of water which is 1.34x10^21 litres. So, for one person to cry that much through all his past lives, he needs to have lived for 1.34x10^21/64 lifetimes, which is still in the order of 10^19, which is more than a quintillion lifetimes.

Based on these descriptions, I would say that the existence of the universe (with all its worlds) where sentient beings live, and the amount of past lives that we have had, is effectively infinite. The past lives recalled by the Buddha are only the last n lives, rather than ALL of it.