How did the United States penal system evolve?

As some of the comments have pointed out, the difference between the US and other countries is not quite as extreme as implied by the question. It's also not really true that the US in general has capital punishment. Only a fraction of the states have death penalties that are actually carried out in practice.

However, it's true that the US has the world's highest incarceration rate and very long prison sentences, which makes it unusual compared to the typical western democracy. There are the following unusual social factors in the US that have affected how criminal justice played out here:

1. The US was expanding into a frontier for much of its history.

2. The south-eastern US had a slave economy for a long time, and the US retained slavery for much longer than, e.g., the UK.

3. If you watch a John Wayne movie, you'll see the tendency of US culture to value individualism, self-reliance, and violence. We also have the right to bear arms enshrined in our constitution.

Because of factor #1, the US has a lot of history of rough frontier justice. Judges and lawyers rode a circuit on horseback. Often no formal court system existed, and extrajudicial punishments such as lynchings were used in order to enforce order. We have, for instance, Placerville, California, which was known as "Old Hangtown," because the judges there during the gold rush would hang anyone who committed a crime. This may sound harsh, but it was actually a step up from extrajudicial lynchings.

Factor #2 means that the US has a longstanding history of racial antagonism and inequality, including a lot of racial violence. In much of the US (not just the southeast) during Reconstruction, it was pretty common for black men to be lynched on some pretext because they didn't submit to a subservient role. (There are for example a lot of old postcards showing grisly lynching scenes.) Today, the US has a lot of black men in prison serving extremely long prison terms for selling crack, while selling powder cocaine (traditionally a drug used by affluent whites) results in short sentences. There is a growing realization in all three branches of government that this kind of thing expresses a legacy of racist justice in the US, which is clearly ultimately traceable back to our history of slavery. I don't think it's a coincidence that we used to be surpassed in our incarceration rate only by apartheid-era South Africa.

Number 3 means that there is much less emphasis on social harmony in the US than there is in cultures like Japan's. Crime is glorified in our popular culture. E.g., young men "sag" their pants, and this is a style that originated in prisons, where prisoners weren't allowed to have belts for security reasons.

Currently, the biggest reason that the US's incerceration rate is so high is that since the Nixon administration we've been pursuing a policy of "war on drugs." It's not really clear to me why the US has followed such an extreme path in this respect and has refused to recognize that it was failing, while a superficially similar culture such as the Netherlands took such a radically different path.

Over the last 30 years crime rates in the US have gone down, for reasons that sociologists can't definitively explain. (E.g., one theory is that many young men used to steal a car as their first significant crime, but that more recently high-tech anti-theft measures have made that harder to do.) However, our prison population has continued to grow because of long sentences and the continuing demand for illegal drugs. The peak in crime 30 years ago was the cause of a law-and-order reaction that has persisted to this day. That reaction has included determinate sentencing and "three-strikes" laws (meaning that a person who commits three felonies gets a life sentence, with the third felony in one well known case being as trivial as stealing a piece of pizza). The US has jury trials due to our heritage of English common law, but juries are required to vote whether or not to convict someone of a crime without knowing in advance whether a conviction would result in a very long determinate prison sentence.

Another unique feature of the US penal system is the extensive use of solitary confinement for long periods of time. I don't know whether there are underlying cultural or historical reasons that can explain this practice, which people in many other countries would consider to be a form of torture.