Why was Hesse south of the Main left out of the North German Confederation?

I'm not sure about southern Hesse specificially, but many of the states in southern Germany simply didn't want to join the new Confederation. The German history magazine "G — Geschichte" writes in an issue about Bismarck (issue of October 2010) on page 18

Deutschland und Frankreich — vier Jahre schon herrschte kalter Krieg zwischen ihnen. Das Königreich im Osten wurde immer mächtiger. Vorläufiger Höhepunkt war 1866 die Gründung des Norddeutschen Bundes. Nur die süddeutschen Staaten fehlen noch zur Einheit. Sie allerdings sperren sich mit Nachdruck. Verpreußung? Nein danke!

Which would translate to

Germany and France — for four years there's been a cold war between the two of them. The Kingdom in the east keeps growing more powerful. The preliminary apex was the foundation of the North German Federation in 1866. Only the southern German states were missing for unification. But those balked [at becoming members] emphatically. Prussification? No, thank you very much!

Because Bismarck (who was basically the driving force) didn't seek to unify the German states out of the kindness of his heart: He wanted to further Prussia's power. Accordingly, Prussia was the decisive power within the new federation. To quote the same magazine as above, from page 29:

Der Norddeutsche Bund ist Preußen — und umgekehrt.

The North German Federation is Prussia — and vice versa.

The constitution of the North German Federation made sure that no decision could be made against Prussia. There were some "Reservatrechte" (reserved rights) for some member states. For example Bavaria could still control its own postal service. But those were more decorative than of any actual impact.

It's probably understandable that the German states weren't too keen to enter that new federation, and some opted to stay out of it as long as they could.

So how would Bismarck go about getting them into his new empire, as well? He'd instigate a war against the old rival, France. For details, you might want to read up on the Ems Dispatch. After this telegram, either side felt gravely insulted by the other, and the French Empire basically had to declare war against the North German Federation to save face.

Another quote from the aforementioned magazine, this time page 21:

Die süddeutschen Staaten sind vertraglich verpflichtet, dem Norddeutschen Bund im Verteidigungsfall beizustehen. Aber selbst wenn sie es nicht wären, würden sie jetzt kämpfen. Denn nun fühlen sich alle nur noch als Deutsche, jetzt geht es gemeinsam gegen Frankreich!

The southern German states are contractually obligated to come to the North German Federation's aid in the case of defense. But even if they weren't obligated, now they would fight. Because now everybody sees themselves only as Germans, now we go together against France!

After the victory over France in the Franco-Prussian War of 1870, the southern states basically ran out of arguments against participating in a unified Germany. And this brings us to the Hall of Mirrors in Versaille, where the German Empire was officially proclaimed on January 18, 1871.

There were two Hesses in 1866, Hesse-Cassel and Hesse-Darmstadt, the Grand Duchy of Hesse.

Hesse-Cassel was annexed by the Kingdom of Prussia in 1866, along with the Kingdom of Hanover, to have a corridor between two groups of Prussian possession. The parts of Hesse-Darmstadt north of the Main were kept by the Grand Duke in 1866 and became part of the North German Confederation led by Prussia. The Grand Duke also kept the parts of Hesse-Darmstadt south of the Main, which remained outside of the North German Confederation.

There were several precedents for this in the history of the German Confederation and of the Holy Roman Empire.

See this question:

https://history.stackexchange.com/questions/35773/how-did-member-states-of-the-holy-roman-empire-justify-only-including-part-of-th[1]

And my answer to this question:

https://history.stackexchange.com/questions/34309/which-european-nation-had-the-most-kings-in-the-18th-century/34372#34372[2]

In 1871 all of Hesse-Darmstadt, still ruled by the Grand Duke, became a state in the German Reich.

https://en.wikipedia.org/wiki/Grand_Duchy_of_Hesse[3]

The answer of Pieter Geerkens explains Bismarck's main diplomatic goals and the various agreements he made to achieve those goals in the peace treaty, agreements which resulted in part of Hesse-Darmstadt being in the North German Confederation and part of it remaining outside for over four years.

On the face of it, simply because the Treaty of Prague stipulated as much:

Article IV

His Majesty the Emperor of Austria recognises the dissolution of the present German Confederation and grants his permission to a new design of Germany without the participation of the Austrian Imperial State. Likewise, His Majesty promises to recognise the narrower federal relationship that the King of Prussia will form north of the line of the River Main, and declares to agree that the German states lying to the south of this line will join each other in a Union, to whom the national connection to the North German Confederation of the close understanding between both will be reserved and that will have an internationally independent existence.

However that is disingenuous. As the overwhelming victor, Prussian King Wilhelm I (with the advice and counsel of Bismarck) is dictating the terms. So with an undenied interest, realized just five years later, of uniting all Germany, why do Wilhelm and Bismarck in 1866 require the exclusion of the (Catholic) South German states? It's complicated:

• Despite being in negotiations to end a current war with the Austrian Empire, Bismarck wants an alliance with Austrian Empire going forward — against France. He just wants them excluded (Note "grants his permission to a new design of Germany without the participation of the Austrian Imperial State." above) from the German Confederation. To this end the terms of the Treaty are very lenient, allowing them to save face.

• Another means of allowing the Austrian Emperor to save face is by coercing them to cede Venetia — the last remaining piece of the Kingdom of Lombardy-Venetia — to France and not Italy.

Article II

In order to execute Article VI of the peace preliminaries agreed in Nikolsburg on 26 July this year, and after His Majesty the Emperor of the French has administrively declared to His Majesty the King of Prussia through his certified messenger at Nikolsburg on 29 July: „qu'en ce qui concerne le Gouvernement de l'Empereur, la Vénétie est acquise á l'ltalie pour lui étre remise á la paix",4 - His Majesty the Emperor of Austria from his side also accedes to this declaration, and grants his permission to the unification of the Lombardo-Venetian Kingdom with the Kingdom of Italy without any other annoying condition, as a liquidation of all debts, which adhering to the ceded territories, will be recognised, in agreement with the stipulations of the Treaty of Zürich.

But Bismarck holds a secret protocol with France that Napoleon III will in turn cede this territory to Italy.

• To guarantee that France (who "supports" Italian unification while continuing to maintain a French garrison in Rome, denying Italy its capital) will keep its secret protocol, Bismarck in turn makes the promise — in exchange also for French neutrality in this just concluded war — that:

that the states south of the Main should have “an internationally independent existence.”

_{Also in A. J. P. Taylor's The Courses of German History (1945), page 124:}

_{In 1866 Bismarck was not ready for war against France, and he agreed to give the German states south of the the Main 'an internationally independent existence'. These states had no power or reality. They were just as much conquered as the states north of the Main. But they survived four more years under the shadowy protection of Napoleon's palsied hand. Bismark was in no hurry. He had to create, almost single-handed the North German Confederation; and the outbursts of national enthusiasm which followed the events of 1866 stirred up his fears of German radicalism.}

Der Nord-Deutsche Bund