Why did the American South agree to the compromise of 1850?

It was impossible (or at least too risky) to deny or postpone California statehood. Their population was swelling because of a gold rush. Their territory was too valuable and too remote to control if the inhabitants were to angry about their request for free statehood not satisfied.

And there was no slaveholding territory ready for (slave) statehood to balance California's acceptance as free state.

So southerners were forced to agree to a statehood for California as a free state, in exchange for the possibility of extending slavery north above the previous 36º 30' limit (nullifying the Missouri Compromise of 1820), and for tightening the fugitive slave act.

The Compromise of 1850 actually undid most of the damage to Southern interests resulting from the Missouri Compromise (of 1820).

The damage was due to the fact that the "compromise" line was 36º 30', the southern boundary of Missouri. That is to say that Missouri was allowed to be a slave state, but all states west of Missouri would be free.

Imagine five tiers of states running south to north including Louisiana, Arkansas, Missouri, Iowa, and Minnesota. The North had three "tiers" of states (excluding Missouri in the middle tier), the South had only two tiers; the northern states would inevitably outnumber the southern.

The reason the South agreed to this in 1820 was because the U.S. had the Louisiana Purchase (the northern tiers) but did not have Texas and California, and the land in between (the southern tiers).

Yes, the South was forced to admit California as a free state, which as it turned out, upset the balance of power in the states. To offset this several major concessions were made to the South.

1) The South quashed the so-called Wilmot Proviso that would have banned slavery in the territories newly acquired from Mexico. This meant that the southernmost tier of states (New Mexico and Arizona) could become slave states.

2) The Federal government assumed the debts of the new state of Texas, after dividing the "old" Texas into Texas and New Mexico, thereby creating the possibility of another slave state (New Mexico). The main reason the South didn't get any further benefits from this was that "Texans" wanted to keep the remainder of the state whole, while other Southerners would have preferred to have it subdivided into at least three or four slave states.

3) The Compromise of 1850 opened the way for a Kansas-Nebraska Act which in theory could have allowed slavery in the Missouri "tier" of states. (They were Kansas, Colorado, Utah, and Nevada, and maybe even California would reconsider.)

4) The South got a version of the Fugitive Slave Law that it wanted.

It actually was not that bad of a deal for the slavery position. Under the previous Missouri Compromise, the Utah Territory (modern-day Nevada, Utah, and Colorado) were guaranteed to come into the Union as Free states. With the new deal, slaves were allowed into the territories, and they would get to vote on whether to have slavery when they entered the union.

Now of course in theory a vote meant those states might not become slave states. However, in practice it was thought by people on both sides that once slavery became established in a territory, it would be very hard to get rid of it politically.

A simple look at the map above will show that the vast majority of unorganized territory in 1849 American was above the Missouri Compromise line. Both sides thought this likely meant the ultimate doom of slavery in the USA. The North was happy to let that happen, and thus liked the status quo, while the South was desperate to find a way to prevent it.

The new higher (more northerly) vote line gave them the potential of 7 new slave states, instead of just 3 (4 if you count the southern part of Cali, which was made part of free California). The provision allowing Texas to optionally split into up to 5 states would have been plenty to keep the slavery-enabling balance given that the remaining (gray) area eventually added up to 10 states.