How do we know numbers of past lives and rebirths?

The Buddhas recollect past abodes or adherences (nivasa) rather than past lives (jiva). This means they recollect each time in the past the mind ignorantly clung to one or more aggregates as 'self'. Refer to SN 22.79, which literally explains this.

Therefore, if it is counted each time the mind clung to something as "I", "me" & "mine", this is counting "pubbe nivasa" (past abodes; not "past lives", which is a mistranslation).

No, it is the number of eons since a certain point of time. ex: number of eons since the Bodhisatta thought to become a Buddha, the number of eons since the Bodhisatta got the blessing to become a Buddha from the Buddha Deepankara and so on.

Except for the countless beginnings and destructions of world systems, there is no beginning of time in Buddhism.

Apparently in Buddhist cosmology, the universe of worlds begins and ends cyclically. That's called an aeon or eon or kalpa. The Visuddhimagga calls this a Maha Kalpa (Great Aeon).

This wikipedia page (without sufficient citations) gives some statistics on the duration of a Maha Kalpa:

Imagine a huge empty cube at the beginning of a kalpa, approximately 16 miles in each side. Once every 100 years, you insert a tiny mustard seed into the cube. According to the Buddha, the huge cube will be filled even before the kalpa ends.

Imagine a gigantic rocky mountain at the beginning of kalpa, approximately 16 x 16 x 16 miles (dwarfing Mount Everest). You take a small piece of silk and wipe the mountain once every 100 years. According to the Buddha, the mountain will be completely depleted even before the kalpa ends.

I found that a tiny mustard seed could have a diameter of around 1mm. If I make it easier by assuming that it is a cube with the sides of 1 x 1 x 1mm, then it has 1 cubic mm of volume.

Meanwhile, a huge empty cube of 16 miles on each side has 17 billion cubic mm of volume. So, if this huge empty cube is filled up at 1 cubic mm per 100 years, then how many years does it take to fill up 17 billion cubic mm of volume?

In my calculation, it would be 1.7 trillion years. So, the description is that this huge empty cube will be filled up before the aeon ends. Currently, physicists estimate that the age of our current universe is 13.82 billion years.

The same page also states:

If you count the total number of sand particles at the depths of the Ganges river, from where it begins to where it ends at the sea, even that number will be less than the number of passed kalpas.

I'm not sure how many sand particles there are at the depth of the Ganges river from the beginning to the end of it, but I guess there must be at least trillions of particles based on this article. So, according to this description, we have had at least trillions of aeons in the past, with each lasting over 1.7 trillion years each.

In the Assu Sutta, it is taught that the amount of tears that we have shed in past lives exceed the volume of the four great oceans.

According to this article, the average amount of tears shed by a human being in a lifetime is 64 litres. And according to this article, the ocean has 321,003,271 cubic miles of water which is 1.34x10^21 litres. So, for one person to cry that much through all his past lives, he needs to have lived for 1.34x10^21/64 lifetimes, which is still in the order of 10^19, which is more than a quintillion lifetimes.

Based on these descriptions, I would say that the existence of the universe (with all its worlds) where sentient beings live, and the amount of past lives that we have had, is effectively infinite. The past lives recalled by the Buddha are only the last n lives, rather than ALL of it.