Which church father first taught the perpetual virginity of Mary?

Ignatius of Antioch is the earliest (circa. 110 AD).

Although may of his writings may be lost, for the writings we do have: when he referred to Mary, he simply called her the Virgin. This is the case whenever he refers to her. If she was not later a virgin, how could some one refer to her in the present tense as a virgin?

Either Bavinck or the O.P. has misread something

The OP states:

Bavinck seems to discount the Protevangelium of James as not being the work of a church father, and for not mentioning Mary's continued virginity after Jesus' birth. He then goes on to identify a wide variety of post-Nicene authors who teach Mary's virginity both during and after the birth.

This statement is the basis for the rest of the OP's question. However, either the OP has misunderstood Bavinck, or Bavinck has badly misunderstood the Protoevangelium (Infancy Gospel) of James. It's true that this gospel is not by a church father, but it is false that it does not mention Mary's continued virginity after Jesus' birth. Quite the contrary, in chs. 19-20 it confirms her continued virginity in very graphic terms:

And Salome said, "As the Lord my God lives, unless I insert my finger and investigate her, I will not believe that a virgin has given birth." And the midwife went in and said, "Mary, position yourself, for not a small test concerning you is about to take place." When Mary heard these things, she positioned herself. And Salome inserted her finger into her body... And Salome went to the child and lifted him up, saying, "I worship him because he has been born a king to Israel." Infancy Gospel of James chs. 19-20

The dating of the Infancy Gospel of James may still place it after the first church fathers began to mention Mary's perpetual virginity, but it should definitely NOT be dismissed for ignoring the issue of Mary's continued status as a physical virgin after birth.

Christians know Mary was a virgin in conceiving and bearing Christ to term. The belief that Mary remained a virgin consists of two additional parts. The first is that she remained a virgin even in the act of giving birth and second is that she remained a virgin thereafter. These two beliefs source to the same book; that is, the Infancy Gospel of James (aka Book of James or Protoevangelium of James) (PoJ). The PoJ was written c170 CE. Origen mentioned it about 220 CE. Clement of Alexandria also mentions it about 200 CE. Augustine, also a “church father”, however, appears to be the first who will clearly clean up, expound, and tie the doctrine together from where the PoJ ends.

Part 1

The first part of the ever-virgin idea (even in birth) was explained in the PoJ that there was no afterbirth, no umbilical cord, no placenta; it was not a normal birth. The baby just appeared. She thus remained a virgin.

And immediately the cloud disappeared out of the cave, and a great light shone in the cave, so that the eyes could not bear it. And in a little that light gradually decreased, until the infant appeared, and went and took the breast from His mother Mary. And the midwife cried out, and said: This is a great day to me, because I have seen this strange sight. And the midwife went forth out of the cave, and Salome met her. And she said to her: Salome, Salome, I have a strange sight to relate to you: a virgin has brought forth— a thing which her nature admits not of. Then said Salome: As the Lord my God lives, unless I thrust in my finger, and search the parts, I will not believe that a virgin has brought forth. (source)

Clement of Alexandria, a “church father” confirms the PoJ as the source of the belief that Mary remained a virgin during birth, even as he denies its veracity.

But, as appears, many even down to our own time regard Mary, on account of the birth of her child, as having been in the puerperal state, although she was not. For some say that, after she brought forth, she was found, when examined, to be a virgin. (source)

In other words, when examined, PoJ says that Mary was still a virgin, remained in the puerperal (childbirth) state. This means it was not a normal human birth with everything that entails. Clement denies she remained in that childbirth state; that is, Christ was born normally and virginity would thus be over. Keep this in mind as Augustine will pick up on this PoJ teaching.

Part 2

The second part of the ever-virgin idea (no sex) means that there were no subsequent brothers or sisters of Christ. There was no sex between Mary and Joseph. The brothers are really step-brothers; this was also explained in the PoJ that Joseph was previously married and had children.

And the priest said to Joseph, You have been chosen by lot to take into your keeping the virgin of the Lord. But Joseph refused, saying: I have children, and I am an old man, and she is a young girl. I am afraid lest I become a laughing-stock to the sons of Israel. (source)

Origen, also a “church father” confirms this book as the source of the ever-virgin after birth by referencing in the book that Jesus' brothers were sons of Joseph and a previous wife saying,

And depreciating the whole of what appeared to be His nearest kindred, they said, “Is not His mother called Mary? And His brethren, James and Joseph and Simon and Judas? And His sisters, are they not all with us?” They thought, then, that He was the son of Joseph and Mary. But some say, basing it on a tradition in the Gospel according to Peter, as it is entitled, or “The Book of James,” that the brethren of Jesus were sons of Joseph by a former wife, whom he married before Mary. (source)

So, we’ve seen that the idea of the ever-virgin during birth and no sex afterwards sources to PoJ, but is mentioned early on by two “church fathers”. One disagrees and one agrees. Here now is Augustine some 200 years later picking up and tying together the two-pronged trail that Mary remained a virgin at birth and after birth.

And if her [Mary] virginity had been marred even in bringing Him [Christ] forth, He would not have been born of a virgin; and it would be false (which God forbid) that He was born of the Virgin Mary (source)

This next quote is from Aquinas who quotes Augustine about the closed gate (no sex thereafter).

It is written (Ezech. 44:2): "This gate shall be shut, it shall not be opened, and no man shall pass through it; because the Lord the God of Israel hath entered in by it." Expounding these words, Augustine says in a sermon (De Annunt. Dom. iii): "What means this closed gate in the House of the Lord, except that Mary is to be ever inviolate? What does it mean that 'no man shall pass through it,' save that Joseph shall not know her? And what is this---'The Lord alone enters in and goeth out by it'---except that the Holy Ghost shall impregnate her, and that the Lord of angels shall be born of her? And what means this---'it shall be shut for evermore'---but that Mary is a virgin before His Birth, a virgin in His Birth, and a virgin after His Birth?" (source)

This interpretation and application that the eastern gate would be shut simply follows the PoJ way that the baby just appears at her side; that is, at her eastern gate, rather than being born normally in the normal manner through the “southern gate”.

So, while the author, whoever it may be, of PoJ is not considered a “church father”, Origen, Augustine, Aquinas, and many other “church fathers” over the centuries continue to trek on its way that the brothers of Jesus were sons of Joseph and a previous wife, that Mary remained in the puerperal state at His birth, and that Jesus was born out from her side. Thus, in this way, she remained a virgin even in the act of giving birth and thereafter.

PS

There is a common misunderstanding that Irenaeus supported the notion of ever-virginity. He didn't.

And if the former [Eve] did disobey God, yet the latter was persuaded to be obedient to God, in order that the Virgin Mary might become the patroness4617 (advocata) of the virgin Eve. And thus, as the human race fell into bondage to death by means of a virgin, so is it rescued by a virgin; virginal disobedience having been balanced in the opposite scale by virginal obedience. http://www.ccel.org/ccel/schaff/anf01.ix.vii.xx.html

Everyone agrees that the virgin Eve did not remain in that state. Yet, people skip that application!

Moreover Irenaeus explains further the nature of the birth. He contrasts those who believe, like the PoJ teaching, that Jesus took nothing from Mary, merely passing through her without afterbirth and umbilical cord with those who like Irenaeus, Clement of Alexandria, Tertullian, Africanus, and others, believed in a normal birth with a normal human birth.

He asks, identifying "of Mary", not "of the ever-virgin",

Why, then, did not God again take dust, but wrought so that the formation should be made of Mary? It was that there might not be another formation called into being, nor any other which should [require to] be saved, but that the very same formation should be summed up [in Christ as had existed in Adam], the analogy having been preserved.

He answers,

1. Those, therefore, who allege that He took nothing from the Virgin do greatly err, [since,] in order that they may cast away the inheritance of the flesh, they also reject the analogy [between Him and Adam]. ... For if He did not receive the substance of flesh from a human being, He neither was made man nor the Son of man; and if He was not made what we were, He did no great thing in what He suffered and endured. ... But Eve was disobedient; for she did not obey when as yet she was a virgin. ... And thus also it was that the knot of Eve’s disobedience was loosed by the obedience of Mary. For what the virgin Eve had bound fast through unbelief, this did the virgin Mary set free through faith.

People wrongly take the same words applied to both Eve and Mary and then based on today's belief fail to apply what is obvious. Yet does not mean perpetual.

As to other allegations about Polycarp, Justin Martyr, and Ignatius also believing in an ever-virgin motif, please note that no quotations from them are provided that say such beliefs. In other words, there's no proof any of them taught or thought that Jesus was born in any way but by a normal human birth.

As with many doctrines in the first few hundred years of the church, they tended not to be explicitly defined until heretics arose forcing the church to define doctrine.

This is why history is scarce on people speaking of this doctrine until the Antidicomarites show up in the mid to late 300s.

With that said, here's the records we have:

• ~100 AD: Ignatius of Antioch
• ~150 AD: Polycarp (disciple of John the Apostle)
• ~160 AD: Justin Martyr

• ~200 AD: Irenaeus

  In the year AD 383, Jerome writes that Ignatius, Polycarp, Justin Martyr, and Irenaeus all “held these same views” of Mary’s perpetual virginity and “wrote volumes replete with wisdom” (in his The Perpetual Virginity of Blessed Mary: Against Helvidius, section 19). No writings from these 4 men survived that unambiguously identifies their belief in this doctrine, but we assume Jerome had access to some of their many works that did not survive until the modern day.

• 248 AD: Origen

  "Mary, as those declare who with sound mind extol her, had no other son but Jesus" [Origen's Commentary on the Gospel of John (Book I), Section 6]

Then as the Antidicomarites show up, we see an explosion in references to the doctrine (after the council of Nicea in 325 AD, as your source noted):

• 354 AD: Hilary of Poitiers

  "If they [the brethren of the Lord] had been Mary's sons and not those taken from Joseph's former marriage, she would never have been given over in the moment of the passion [crucifixion] to the apostle John as his mother, the Lord saying to each, 'Woman, behold your son,' and to John, 'Behold your mother' [John 19:26-27], as he bequeathed filial love to a disciple as a consolation to the one desolate" [Hilary's Commentary on Matthew 1:4]

• 360 AD: Athanasius

  Identifies Mary as "Mary Ever-Virgin" in his Discourse 2 Against the Arians, Section 70

• 373 AD: Ephrem

  "Because there are those who dare to say that Mary cohabited with Joseph after she bore the Redeemer, we reply, 'How would it have been possible for her who was the home of the indwelling of the Spirit, whom the divine power overshadowed, that she be joined by a mortal being, and gave birth filled with birthpangs, in the image of the primeval curse?'" [Ephrem's Commentary on Tatian's Diatessaron]

• ~375 AD: Basil of Caesarea

  "...the lovers of Christ do not allow themselves to hear that the Mother of God ceased at a given moment to be a virgin..." [Basil’s Homily: On the holy generation of Christ 5; PG 31, 1468 B]

• 375 AD: Epiphanius

  "For I have heard from someone that certain persons are venturing to say that [Mary] had marital relations after the Savior’s birth. And I am not surprised. The ignorance of persons who do not know the sacred scriptures well and have not consulted histories, always turn them to one thing after another, and distracts anyone who wants to track down something about the truth out of his own head.” [The Panarion of Epiphanius of Salamis: De fide. Books II and III, page 620, 7.1]

• 383 AD: Jerome

  In his The Perpetual Virginity of Blessed Mary: Against Helvidius he gives a long, full biblical defense of Mary's perpetual virginity, noted in earlier sections in this answer.

• 386 AD: Didymus the Blind

  "Mary... remained always and forever an immaculate virgin" [Didymus's The Trinity 3:4]

• 388 AD: Ambrose of Milan

  Identified prophecy of Ezekiel 44:2 as proof of Mary's perpetual virginity in his De Institutione Virginum 8.52

• 401 AD: Augustine

  "A Virgin conceiving, a Virgin bearing, a Virgin pregnant, a Virgin bringing forth, a Virgin perpetual. Why do you wonder at this, O man?" [Augustine, Sermons 186:1]

And so on, this wiki continues the belief through to the modern era including the reformers.