This was a matter of controversy in the early church; the early 3rd century appears to represent the climax of the debate.

Earliest accounts

The Apostolic Fathers are silent on the matter. The earliest surviving sources to broach this topic date to the late 2nd century:

• The Gospel of James (the "Protoevangelium") & Clement of Alexandria are sometimes used to argue in favor of the perpetual virginity of Mary
• Hegesippus & Victorinus are sometimes used to argue against it

However, none of these sources are explicit and, the most popular of the 4 (on this topic), the Gospel of James, is a problematic text:

• It is pseudepigraphal (the person writing it claimed to be someone he was not)
• The text was condemned by Innocent I in the 5th century
• The text was written long after the eyewitnesses to Jesus' (and Mary's) life were gone. This does not mean it contains no historical truth, but it should not be treated as a primary source as the 4 canonical Gospels are.

--

Third Century

It is in the 3rd century that we see explicit teachings in favor of and opposed to the perpetual virginity of Mary.

Among the most prominent in favor:

Hippolytus

But the pious confession of the believer is that, with a view to our salvation, and in order to connect the universe with unchangeableness, the Creator of all things incorporated with Himself a rational soul and a sensible body from the all-holy Mary, ever-virgin, by an undefiled conception, without conversion, and was made man in nature (Against Beron & Helix, Fragment VIII)

Origen

Origen acknowledges the view and the reasonable nature of its proponents, without fully endorsing it:

For if Mary, as those declare who with sound mind extol her, had no other son but Jesus (Commentary on John 1:6)

They thought, then, that He was the son of Joseph and Mary. But some say, basing it on a tradition in the Gospel according to Peter, as it is entitled, or The Book of James, that the brethren of Jesus were sons of Joseph by a former wife, whom he married before Mary. Now those who say so wish to preserve the honour of Mary in virginity to the end (Commentary on Matthew 10:17)

And opposed:

Tertullian

although she was a virgin when she conceived, she was a wife when she brought forth her son (On the Flesh of Christ, ch. XXIII)

--

Later Years

By the 4th century, the preponderance of surviving records favor the perpetual-virginity view, though the matter would continue to be debated for several generations more (see here). The Second Council of Constantinople in AD 553 officially anathematized those who opposed the belief that Mary was a perpetual virgin.

--

Conclusion

The early church debated this topic; consensus on the matter was a post-Nicene development.

Virginity (virginitas)

before birth (ante partum)

St. Justin Martyr, c. 100-165 A.D. (Apol., I.):

The words “Behold, a virgin shall be with child” [Is. 7:14] mean that the virgin shall conceive without fleshly commerce. For had she admitted such commerce, she would no longer be a virgin. But the power of God effected that she conceived as a virgin.

during birth (in partu)

St. Ambrose, 340-397 A.D. (De Instit. Virg. [On the Virgin and on the Perpetual Virginity of Holy Mary], VIII, n. 52):

The prophet Ezechiel [44:2] says that he saw the building of a city upon a very high mountain. The city had many gates. Of these one is described as shut. What is this gate but Mary? And shut because a virgin. Mary, then, is the gate through which Christ came into this world, when he was shed forth by a virginal birth, without loosing the bars of virginity. The inclosure of purity remained unscathed, and the seals of integrity were kept inviolate, as He went forth from the virgin.… A good gate is Mary, that was closed, and was not opened. By her Christ passed, but He opened not.

after birth (post partum)

St. Jerome, c. 341-420 A.D. (Contra Helvid. [The Perpetual Virginity of Blessed Mary], 19):

[St. Joseph] was Mary’s protector rather than her husband, and like her, led a celibate life.
_{Mariæ custos potius fuit quam maritus; relinquitur, virginem eum mansisse cum Maria.
lit.: "He was more Mary's guardian than husband; hence, it follows that he remained a virgin with Mary.}

^{St. Joseph's remaining a virgin means Our Lady did, too.}

^{Pohle-Preuss, Mariology, pt. 2, ch. 1, §3 "Mary’s Perpetual Virginity"}