How can the existence of other species of the genus Homo be reconciled with Christian belief in the uniqueness of humankind? (Catholic perspective)

Starting with the question title, I will quote the definition of genus from wikipedia:

A genus (/ˈdʒiːnəs/, pl. genera) is a taxonomic rank used in the biological classification of living and fossil organisms in biology. In the hierarchy of biological classification, genus comes above species and below family.

Therefore, the existence of other species of genus Homo poses for Christian belief in the uniqueness of humankind the same problem as the existence of other species of family Hominidae or of subfamily Homininae: none at all.

Moving on to the question text, I will focus on the assertion "They weren't mere brutes, as archaeological research has shown, but they weren't humans, either."

The case of any extinct species of genus Homo, be it Neanderthal or whatever, is very simple: either they were human beings from the theological viewpoint or they were not. I.e., either they had a spiritual soul or they did not.

Regarding this subject, Catholic doctrine says only that all human beings (again, from the theological viewpoint, i.e. having a spiritual soul) descend patrilineally from the first man, biblical Adam. Therefore:

• If Neandies had a spiritual soul, then biblical Adam was a common patrilineal ancestor of both Neanderthals and Sapiens.

• If Neandies did not have a spiritual soul, then biblical Adam lived after the "Neanderthal line" diverged from the "Sapiens line", and belonged to the latter.

And that's it.

Now, the question of whether Neandies had a spiritual soul is definitely outside the scope of Catholic doctrine. Of course, if they had written a derivation of calculus in the walls of a cave, then it would be clear that they had abstract reasoning and therefore a spiritual soul. But inferring from archeological remains whether they had abstract reasoning and true human speech (as described e.g. by Chomsky's universal grammar) and therefore a spiritual soul is a field in which you can be quite sure the Catholic Magisterium will never enter.

I would say that of the Christians that have an opinion on the subject (it's not really a theological point that comes up often) they seem to consider them human as well. Consider this article

Based on the comparison of modern human mt DNA and that taken from the Neanderthal, evolutionists have argued that the "Neanderthal line" diverged from the line of "hominids" leading to modern humans about 600,000 years B.P. without contributing mt DNA to modern Homo sapiens populations. This strongly implies that Neanderthals were a different species from modern humans.

However, the above noted interpretation is not scientifically justified. Lubenow (1998) has pointed out that the use of a statistical average of a large modern human sample (994 sequences from 1669 modern humans) compared with the mt DNA sequence from one Neanderthal is not appropriate. Furthermore, the mt DNA sequence differences among modern humans range from 1 to 24 substitutions, with an average of eight substitutions, whereas, the mt DNA sequence differences between modern man and the Neanderthal specimen range from 22 to 36 substitutions, placing Neanderthals, at worst, on the fringes of the modern range.

The uniqueness of humanity is solely about us being image bearers of YHWH. Current human physical traits differ enough that some have argued that certain homi sapiens aren't actually human (see chattel slavery, and other forms of similar oppression). But now most would say that's ridiculous. So your question is perhaps more relevant to our recent past, and even current events, than many think.

But to address your question, any relative of h*m* sapien that YHWH made image bearers would be considered...image bearers, and thereby, "unique". Maybe at the root of your question is, "when did YHWH first make something His image bearer, and did that include every ancestor?"