How could we have so many past lives without reaching some degree of wisdom or enlightenment?

It seems to me if had an infinity of past lives we should have reached wisdom and buddahood at some point.

Indeed.

The Buddha himself did not teach about "past lives". The words commonly translated as "past lives" in the original scriptures are "pubbe nivasa". The word "nivasa" means "homes", "abodes" or even "adherences" (in the related word "nivesa").

A "nivasa" is when one or more of the five aggregates (body, feelings, perception, mental formations & sense consciousness) are attached to as "self".

Therefore, in this life, each mind has clung to aggregates as "self" infinite times. The relevant scripture, properly translated, says:

At Savatthi. Bhikkhus, those ascetics and brahmins who recollect their manifold past abodes (pubbe nivasa) all recollect the five aggregates subject to clinging or a certain one among them. What five?

When recollecting thus, bhikkhus: ‘I had such form in the past,’ it is just form that one recollects. When recollecting: ‘I had such a feeling in the past,’ it is just feeling that one recollects. When recollecting: ‘I had such a perception in the past,’ it is just perception that one recollects. When recollecting: ‘I had such volitional formations in the past,’ it is just volitional formations that one recollects. When recollecting: ‘I had such consciousness in the past,’ it is just consciousness that one recollects.

Therefore, bhikkhus, any kind of form whatsoever … Any kind of feeling whatsoever … Any kind of perception whatsoever … Any kind of volitional formations whatsoever … Any kind of consciousness whatsoever, whether past, future, or present, internal or external, gross or subtle, inferior or superior, far or near, all consciousness should be seen as it really is with correct wisdom thus: ‘This is not mine, this I am not, this is not my self.’

This is called, bhikkhus, a noble disciple who dismantles and does not build up; who abandons and does not cling; who scatters and does not amass; who extinguishes and does not kindle.

And what is it that he dismantles and does not build up? He dismantles form and does not build it up. He dismantles feeling … perception … volitional formations … consciousness and does not build it up.

And what is it that he abandons and does not cling to? He abandons form and does not cling to it. He abandons feeling … perception … volitional formations … consciousness and does not cling to it.

And what is it that he scatters and does not amass? He scatters form and does not amass it. He scatters feeling … perception … volitional formations … consciousness and does not amass it.

And what is it that he extinguishes and does not kindle? He extinguishes form and does not kindle it. He extinguishes feeling … perception … volitional formations … consciousness and does not kindle it.

https://suttacentral.net/sn22.79/en/bodhi

It is clearly noticeable how the answers provided to your question by the common Buddhist are contrary to what is written above. The common Buddhist refers to infinite past lives yet the above says the Noble Disciple "dismantles, does not build up, does not amass, does not kindle".

Other common false translations found in this type of topic are "sentient beings" and "transmigration".

The Pali word falsely translated as "sentient being" is "satta". "Satta" merely means "a being" and, similar to "nivasa", refers merely to an attachment or a view, as explained in SN 23.2 & SN 5.10.

The Pali word falsely translated as "transmigration" is "saṃsāra". "Saṃsāra" merely means "continuance" and, similar to "nivasa", refers merely to the continuance of attachment, as explained in SN 22.99, as follows:

Just as a dog, tied by a leash to a post or stake, keeps running around and circling around that very post or stake; in the same way, an uninstructed, run-of-the-mill person — who has no regard for noble ones, is not well-versed or disciplined in their Dhamma; who has no regard for people of integrity, is not well-versed or disciplined in their Dhamma — assumes form to be the self, or the self as possessing form, or form as in the self, or the self as in form.

He assumes feeling to be the self...

He assumes perception to be the self...

He assumes (mental) fabrications to be the self...

He assumes consciousness to be the self, or the self as possessing consciousness, or consciousness as in the self, or the self as in consciousness.

He keeps running around and circling around that very form... that very feeling... that very perception... those very fabrications... that very consciousness. He is not set loose from form, not set loose from feeling... from perception... from fabrications... not set loose from consciousness. He is not set loose from birth, aging, & death; from sorrows, lamentations, pains, distresses, & despairs. He is not set loose, I tell you, from suffering & stress.

Why false translations occur was explained by the renowned Thai Buddhist monk Bhikkhu Buddhadasa, as follows:

People language is used by the ordinary people who don't understand Dhamma very well and by those worldly people who are so dense that they are blind to everything but material things. Then, there is the language which is spoken by those who understand reality (Dhamma), especially those who know and understand reality in the ultimate sense. This is another kind of language.

People who are blind to the true reality (Dhamma) can speak only people language, the conventional language of ordinary people. On the other hand, people who have genuinely realized the ultimate truth (Dhamma) can speak either language. They can handle people language quite well and are also comfortable using Dhamma language, especially when speaking among those who know reality, who have already realized the truth (Dhamma). Amongst those with profound understanding, Dhamma language is used almost exclusively, unfortunately, ordinary people can't understand a word. Dhamma language is understood only by those who are in the know. What is more, in Dhamma language it isn't even necessary to make a sound. For example, a finger is pointed or an eyebrow raised and the ultimate meaning of reality is understood. So, please take interest in these two kinds of language--people language and Dhamma language.

No Religion

More can be read, here: Two Kinds of Language

It seems to me if had an infinity of past lives we should have reached wisdom and buddahood at some point. How couldn't it be the case since infinity guarantees every possible outcome happens? What would a buddhist ansnwer to me?

Well, the key point is highlighted above. If one places a time limit "cap" on that "at some point" (ie limit it to the past or present moment), then it wouldn't be called "infinity" anymore. So, "at some point" could mean at any time, depending on how hard one's willing to cultivate the Dhamma. It could be tomorrow, in 7 days, 7 weeks, 7 months, ... 7 aeons, or... infinity!

Now, if anyone would develop these four frames of reference in this way for seven years, one of two fruits can be expected for him: either gnosis right here & now, or — if there be any remnant of clinging/sustenance — non-return.

"Let alone seven years. If anyone would develop these four frames of reference in this way for six years... five... four... three... two years... one year... seven months... six months... five... four... three... two months... one month... half a month, one of two fruits can be expected for him: either gnosis right here & now, or — if there be any remnant of clinging/sustenance — non-return.

"Let alone half a month. If anyone would develop these four frames of reference in this way for seven days, one of two fruits can be expected for him: either gnosis right here & now, or — if there be any remnant of clinging/sustenance — non-return.

"'This is the direct path for the purification of beings, for the overcoming of sorrow & lamentation, for the disappearance of pain & distress, for the attainment of the right method, & for the realization of Unbinding — in other words, the four frames of reference.' Thus was it said, and in reference to this was it said."

That is what the Blessed One said. Gratified, the monks delighted in the Blessed One's words. ~~ MN 10 ~~

As with all things religious and philosophical, it's a good idea to reach for the metaphysical point, and avoid getting hung up on the physical logistics of the idea. The point here is to deconstruct relationship roles that we would otherwise be blind to. For example, the relationship of child to parent is a distinct identity role that we naturally think of as immutable; we are our parent's children, and questioning that fact (and the roles it entails) seems absurd on the face of it. Suggesting that our parents may have been our children in a previous life, however, forces us to see the parent-child relationship as contingent — a dependent arising of this lifetime — and allows us to separate ourselves from our role as child.

Whether or not reincarnation as such exists, this point is valid. If we want liberation then these kinds of role-relationships must be rendered contingent so we can free ourselves from them. The mind-puzzle of how this could possibly be is fun, I suppose, but it is anti-liberation: the ego focusing on a physical problem because it is having difficulty with the metaphysical one (in much the same way that one might kick a malfunctioning TV, because learning how to actually fix a TV is too much trouble).

You should perhaps be more specific in your question, heres how it sounds to me: How can I have so many past lives and not already be enlightened? Well, how do you know youre not already? If you are already enlightened you could have hidden knowledge from yourself that would be too painful or hard to comprehend right now so you are being willfully ignorant and waiting until the conditions are right to fully awaken (like a bodhisattva does). Whether or not you remember tasting nibanna does not matter. I personally dont think you have to believe in the vast number of your own past lives. Even if you could accept as fact one past life and recall it and learn from it, this would be benficial.

Your problem sounds more like doubt. You doubt that the people around you are connected to your past lives? I would propose that you do an experiment. Can you do brahmavihara meditation? We are supposed to recall acquaintances and family and different people in our lives to aid in cultivating those feelings of love. The instructions to do this always warn to choose carefully who you bring to the mind because your attraction to them or the feelings of hurt or victimization by them could interfere. So usually that person we bring to the mind in the category of “the adversary” the person to whom it is initially difficult for us to give our metta.

Who is that person to you?? Why do you have so much melodrama with this person? Sex and love and all that usually complicates relationships sooo much that we are left barely even able to muster up the metta to give to the person we loved so much at one point and who now is like an enemy. This is the person who can probably teach you the most about your past life your “sweet karmic enemy”

Infinities can be one-way. Consider the negative numbers.

What the Buddha said is that there is no "first point".

SN15.19:1.3: No first point is found of sentient beings roaming and transmigrating, hindered by ignorance and fettered by craving.

And with the ending of ignorance, one does not return.

AN2.36:4.7: When they pass away from there, they are non-returners, not coming back to this state of existence.

Non-returners (anāgāmi) and arahants do not return. They have left behind the negative infinity of suffering.

It is said to have taken the last Buddha more than four uncountable time periods from the time he received confirmation from an earlier Buddha to become a Buddha. That's already a fairly unfathomably long period of time, and he was of course a very special, rare exception to the norm. Even among Buddhas his journey is said by the commentaries to have been relatively short (it can take up to more than eighty uncountable periods, iirc).

Becoming enlightened isn't just something one falls into accidentally. It is beyond all forms of worldly wisdom and requires exceedingly special circumstances to occur. Even worldly insight into reality that might help us reach enlightenment comes and goes and is replaced by ignorance and delusion, so just because we may have at times gained worldly wisdom in the past does not mean we will eventually become enlightened as a result.

From SN 15.3:

“... What do you think? Which is more: the flow of tears you’ve shed while roaming and transmigrating for such a very long time—weeping and wailing from being united with the unloved and separated from the loved—or the water in the four oceans?”

“As we understand the Buddha’s teaching, the flow of tears we’ve shed while roaming and transmigrating is more than the water in the four oceans.”

“Good, good, mendicants! It’s good that you understand my teaching like this. The flow of tears you’ve shed while roaming and transmigrating is indeed more than the water in the four oceans. For a long time you’ve undergone the death of a mother … father … brother … sister … son … daughter … loss of relatives … loss of wealth … or loss through illness. From being united with the unloved and separated from the loved, the flow of tears you’ve shed while roaming and transmigrating is indeed more than the water in the four oceans.

Why is that? Transmigration has no known beginning. … This is quite enough for you to become disillusioned, dispassionate, and freed regarding all conditions.”

It's true that according to the sutta above, you have been reborn innumerable times, with no known beginning to suffering.

But usually, people take this to mean that the very same consciousness that is reading this sentence has been reborn many times and experienced all that has been experienced through this body since birth.

However, according to MN 38, this is not the case:

The Blessed One then asked him: “Sāti, is it true that the following pernicious view has arisen in you: ‘As I understand the Dhamma taught by the Blessed One, it is this same consciousness that runs and wanders through the round of rebirths, not another’?”

“Exactly so, venerable sir. As I understand the Dhamma taught by the Blessed One, it is this same consciousness that runs and wanders through the round of rebirths, not another.”

“What is that consciousness, Sāti?”

“Venerable sir, it is that which speaks and feels and experiences here and there the result of good and bad actions.”

“Misguided man, to whom have you ever known me to teach the Dhamma in that way? Misguided man, have I not stated in many ways consciousness to be dependently arisen, since without a condition there is no origination of consciousness? But you, misguided man, have misrepresented us by your wrong grasp and injured yourself and stored up much demerit; for this will lead to your harm and suffering for a long time.”

Consciousness is always changing and it is dependently originated. However, it may appear to be the same just as the water stream of a river appears to be the same, when in fact, the droplets of water in it are constantly changing. Please see this answer.

So, yes - you have been reborn innumerable number of times and have shed so much tears mourning the deaths of innumerable loved ones. But who are "you"? You are not the same consciousness that moves through the births.

"You" refers to the self i.e. the thought or idea of the self. It's not a soul. It's simply the thought or idea or view of the self in the mind. It's not the same common self for every person. It's also not the same thought that exists at all moments in time - it's dependently originated.

When there is the thought of the self, there is the objectification and classification (papanca) of everything else that is perceived as non-self, relative to its relationship to the self. This causes the arising of craving and suffering. This is elaborated in this answer.

So, there is definitely rebirth - not the rebirth of a permanent consciousness or soul, but rather the rebirth of the mental idea of the self, and together with it, the rebirth of suffering.

So, the Buddha looks at YOU - yes, that's right - YOU - the SELF - and tells you that you have been reborn innumerable number of times, with no known beginning. You (the SELF) have also shed oceans of tears, mourning the deaths of innumerable loved ones of your's. This to me, is definitely proof that there is rebirth in Buddhism, just not the rebirth that you may have imagined.

And who says that you have not reached some level of wisdom? The very fact that you are writing this question or reading this answer, shows that you have at least enough wisdom to consider that you should seek the end of suffering.

As I understand, the Buddha meant that we collectively had gone through an uncountable number of existances, in all shapes and forms. From this mass of existances, some were able to attain Enlightenment (thereby arising as the Buddhas and their students) and some weren't.

Wisdom and knowledge is definitely accumulated, but not everyone can take advantage of it. The reason for that is karma, which is the law of self-reinforcing tendencies. If someone steps on the wrong path, they create circumstances that lead further and further away from those situations where the wisdom is known to manifest -perhaps delaying or making it harder to get out of that loop.