When and where does the statement, "Christ paid the penalty for our sins" first appear?

The theological term for the concept you are thinking of, namely that Christ was payment for our sins and so took upon Himself the fullness of punishment due to sin, is known as penal substitution (as you identify). Penal substitution is largely originated in the Reform Movement. It is generally agreed that no Church father taught penal substitution, but rather favored the ransom theory of atonement intermixed with the moral influence view. Several Reform movement thinkers and those applying to the penal substitution theory have attempted to attribute the source of their thinking to certain Church fathers, though they have been unsuccessful in doing so. It could be said that the roots for penal substitution were birthed by St. Anselm (though the theory itself can hardly be attributed to him.) St. Anselm sought to modify the traditional ransom theory of atonement. St. Anselm ultimately contributed what is known as Anselmian satisfaction, which is the theory that the satisfaction that God the Father saw in Christ as an honorable and perfect man compensated for the due punishment that man deserved.

The honor taken away must be repaid, or punishment must follow.

This sense of atonement illuminated upon the ransom theory of atonement; Anselm held that Christ's sacrifice was not appeasing as a ransom to Satan (as traditional ransom theory held) but rather to God. And upon this theory St. Thomas Aquinas only further modified the kinks. In progression from this, according to St. Thomas Aquinas, Christ not only fulfilled a debt of honor but also a debt of moral justice. Mankind not only fell in their honor, but also offended God by disobeying Him. So it is that Aquinas concludes the following;

Christ bore a satisfactory punishment, not for His sins, but for ours.

Aquinas does not intend this to mean that Christ bears upon Himself a penal punishment for sin (punishment that serves a lawful purpose accorded to certain individuals, an example being eternal damnation), but rather that He bears upon Himself a satisfactory punishment (punishment that serves a restorative purpose due to its penitent nature, and that's merits can be shared).

If we speak of that satisfactory punishment, which one takes upon oneself voluntarily, one may bear another's punishment... If however, we speak of punishment inflicted on account of sin, inasmuch as it is penal, then each one is punished for his own sin only, because the sinful act is something personal. But if we speak of a punishment that is medicinal, in this way it does happen that one is punished for another's sin.

It is generally understood that the punishment Christ went through, the satisfactory punishment, was death and suffering, the immediate and unavoidable consequences of original sin. For Aquinas recognized that Christ could not merely become perfect and eternal man in order to justify man, but rather that the completion of salvation had to include, because of fallen man's evils, some sort of necessary punishment of a penitent nature for original sin itself (similar to the penitent punishment that believers go through in order to be purified of personal sin). Christ's justifying man through the indwelling perfection of His being had to include submitting in perfect penance for the original sin of man (and to do so Christ entered the satisfactory punishment of original sin, being death). Notice that this punishment is not an appeas*m*nt of an angry God, but rather a part of Christ perfecting man. Aquinas firmly believed that the nature of such punishment was medicinal, working for a Higher Purpose of Salvation, and not the motivation or origin of Salvation itself, which was always essentially a mystery and certainly not 'legalistic'.

Nevertheless, following this strain of thought Martin Luther held that Christ not only satisfies God the Father in His perfection but also bears mankind's penal punishment on the cross.

When the merciful Father saw that we were being oppressed through the Law, that we were being held under a curse, and that we could not be liberated from it by anything, He sent His Son into the world, heaped all the sins of men upon Him, and said to Him: "Be Peter the denier; Paul the persecutor, blasphemer, and assaulter; David the adulterer; the sinner who ate the apple in Paradise; the thief on the cross. In short, be the person of all men. And see to it that You pay and make satisfaction for them.

Similar thinker John Calvin states the following:

God in his capacity as judge is angry toward us. Hence as expiation must intervene in order that Christ as priest may obtain God's favor for us and appease his wrath.

We must, above all, remember this substitution, lest we tremble and remain anxious throughout life — as if God’s righteous vengeance, which the Son of God has taken upon himself, still hung over us.

And in like manner, Puritan John Owen states

For to make satisfaction to God for our sins, is is required only that He undergo the punishment due to them; for that is the satisfaction required where sin is the debt.

John MacArthur even more radically states

God was punishing his own Son as if He had committed every wicked deed done by every sinner who would ever believe.

Note that the concentration has moved from the focus being on Christ perfecting mankind and facing consequentially a necessary punishment to Christ solely becoming man to appease the wrath of God, and implicitly taking upon Himself not only satisfactory punishment but penal punishment also. This is because the Reform thinkers conjoined the distinct types of punishment that Aquinas rather firmly tried to keep separate. It is in light of this Reformed theology that penal substitution supporters live today. Predominant denominations that use the catchphrases 'Jesus died as payment for our sin' include most Protestant denominations, most heavily among them being Lutherans and Methodists.