What factors prevented the legality of secession being pursued via the courts prior to the outbreak of the Civil War?

No judicial but legislative ruling required to make secession legal (at least as the first step).

Secessionists could introduce secession bill to the Congress, and if they did, most likely they could get peaceful separation (considering that Corwin Amendment passed Congress even without votes of seven seceding states).

If bill failed, they could bring the case against Congress decision to the court.

If bill passed but president still don't recognize secession, they could try to impeach him or bring the case against executive branch to the court.

Without this congressional attempt, they did not have case.

The main problem that secessionists would have faced in court is that the question had been settled over 40 years before the Civil War: secession was illegal.

The only argument that supports unilateral secession, where a state just decides to leave and that's sufficient, is called "Compact Theory". The theory is that the US is not a single nation, but a "compact" between Sovereign states, somewhat like a League of Nations. Any nation can unilaterally dissolve a treaty: likewise, under Compact Theory any state could dissolve its participation in the union.

Compact Theory had a long and distinguished history in American politics. Jefferson was a proponent, and at least for a while so was Madison. But Compact Theory was rejected by the US Supreme Court three times before 1820:

• Chisholm v. Georgia (1793)
• Martin v. Hunter's Lessee (1816)
• McCulloch v. Maryland (1819)

So there was no legal leg for unilateral secession. (It's interesting to note that in all 3 cases, the sitting court was majority-Southern.) If secessionists wanted to pursue a strategy of "legal" secession, it would have to be cooperative, "consensual" secession, agreed to by both parties. The only conceivable mechanism for that would probably be the formula from the Articles of Confederation and Perpetual Union: "such alteration be agreed to in a Congress of the United States, and be afterwards confirmed by the legislatures of every State." No way in hell would the legislatures of EVERY state agree to it.

And anyway, that process would be agonizingly slow. Measured in years. The secessionists were men of action who needed to strike while the iron was hot. As Oldcat writes, they were not interested in a court case, nor in a protracted legislative process. They needed to secede before anyone could stop them, and then make it stick by force of arms.

It's also worth noting that they (or most of them) thought they were too big to lose. The Confederacy was an area about the size of Spain + Italy + France + Germany + Poland PUT TOGETHER. No one thought Federal forces could conquer and hold an area that large: British and French military observers told their governments that the North would ultimately lose the war, the only question being how long it would take them to accept reality.

So the factors for the secessionists are:

1. No legal basis.
2. Necessity for a bold stroke.
3. Strong expectation they would win in any military follow-up.

Nobody was interested in a court case. Those who thought it was undoable had no reason for a suit, and those who wanted secession needed it as a bold stroke to rally the more undecided Southern factions to their banner. Secession had failed in 1832 and 1850 once people took the time to consider the matter. Secessionists in 1860 needed for it to be a reality before anyone could back down. They thought that most of the moderate Southerners would join them rather than use force on them, and for the most part this was so.