What is the difference between "ransom theory" and "penal substitution"?

It is the same difference as there is in modern law between a civil lawsuit and a penal lawsuit.

Ransom theory treats the enmity between God and men as if it were a civil lawsuit, i.e. a legal contention which can only be resolved through the payment of damages, here a "ransom". Men owe God a ransom (damages) in order to compensate for their sins (which of course they are unable to pay) and Christ "foots the bill" by dying on the cross.

Penal substitution treats the same enmity between God and men as if it were a penal lawsuit, i.e. one which will lead to a punishment for the guily party (here all men). Christ is a substitute for men as he takes on himself the penalty for the sins of mankind, which would otherwise have fallen upon men, by dying on the cross.

Both are metaphors (parables), of course. They are not mutually exclusive as they can be more easily understood by different categories of people. For example, in most tribal societies, there is no penal justice because there is no state. In such a context, even murder is a civil matter, to be decided by tribal elders, council, etc. For people of such background, the Ransom theory is more easily understandable than the Penal one. For us who live in a society where penal justice is more prominent than its civil counterpart (how many civil law shows are there on TV?), it is probably the opposite.

This question is a bit old, but there is an omission in the only answer. fi11222's answer states that both ransom and penal substitution theories of atonement view God as needing payment of some form. I agree with the description of penal substitution, but I believe the orthodox ransom theory generally teaches that the ransom was paid to death and not God the Father; this being a key difference between the two theories.

Wikipedia ("Ransom theory of atonement") says that the theory teaches the ransom was "paid to Satan or to death itself, in some views paid to God the Father".

The Eastern Orthodox church would teach that viewing the ransom paid to Satan or God the Father to satisfy him is wrong and perhaps heretical. I know that Protestants usually vigorously affirm that the debt was paid to God the Father, but I don't know how Catholics view it.

To give some historical perspective, let's hear from the church fathers themselves (taken from Christian Classics Ethereal Library, "De Inc. 4.20"):

St. Athanasius:

He might offer the sacrifice on behalf of all, surrendering His own temple to death in place of all, to settle man's account with death and free him from the primal transgression.

I believe the above quote is from his work, "On the Incarnation": a great read if you are interested in this sort of thing.

St. Gregory of Nazianzus:

To whom was that blood offered that was shed for us, and why was it shed? I mean the precious and glorious blood of God, the blood of the High Priest and of the Sacrifice. We were in bondage to the devil and sold under sin, having become corrupt through our concupiscence. Now, since a ransom is paid to him who holds us in his power, I ask to whom such a price was offered and why? If to the devil, it is outrageous! The robber receives the ransom, not only from God, but a ransom consisting of God himself. He demands so exorbitant a payment for his tyranny that it would have been right for him to have freed us altogether. But if the price is offered to the Father, I ask first of all, how? For it was not the Father who held us captive. Why then should be blood of His only begotten Son please the Father, who would not even receive Isaac when he was offered as a whole burnt offering by Abraham, but replaced the human sacrifice with a ram? Is it not evident that the Father accepts the sacrifice not because he demanded it or because He felt any need for it, but on account of economy: because man must be sanctified by the humanity of God, and God Himself must deliver us by overcoming the tyrant through His own power, and drawing us to Himself by the mediation of the Son who effects this all for the honor of God, to whom He was obedient in everything... What remains to be said shall be covered with a reverent silence. (In sanctum Pascha, or. XLV, 22', P.G., t 36, 653 AB, quoted in Lossky, Mystical Theology, p. 153.)

The important bit of St. Gregory's quote is where he says "the Father accepts the sacrifice not because he demanded it or because He felt any need for it, but on account of economy".