score:3
Here are some rough economic facts for a start. The North had four times the GDP of the South. The North had 22 million people. The South had 9.5 million people, of which just under 4.0 million were slaves, so 5.5 million white people. If you divide the white populations, you get a four to one North/South ratio, the same as the GDP ratio. This means that the northern and southern per capital GDPs were equivalent, but only if you count the nearly 4.0 million black slaves as property, and their production as part of white per capita GDP. If you divide the South's GDP by all 9.5 million people, including blacks, the southern standard of living was on average, 61% of the northern.
It follows therefore, that the average white southerner who didn't own any slaves was much worse off than the average white northerner (who also didn't own any slaves). The second thing is the "nine tenths of the slaveholders" who had ONE slave were barely better off than the poor white farmers that had none.
The last question is how much Mr. Nathan Johnson's (northern) standard of living was below that of the average white northerner. Presumably, there was some "discount," but probably not all the way down to "61%." It made sense that he was better off than southern whites who owned 0 or even one slave. It was the large plantation owners with tens or hundreds of slaves that were wealthy. The South was a very unequal society, with 1%-2% of the people at the very top, and the rest below that of the northern "average."
Source: Gary Becker, "The Economics of Discrimination"