score:12
Indeed, in a somewhat similar situation, in the 15th century Italian Condottieri such as Sforza, Braccio da Montone and Malatesta did use their mercenary armies to seize control of city-states and become dukes and lords. However, in 17th century Northern Europe such a development was more or less unthinkable. The reason is, I think, that the power in 17th century states had a different source of legitimacy - it was based on more than naked coercion, but on some variation on the divine right of kings/dukes/princes etc. The more complex structure of the economy also played an important role.
Also, one can take control of a city-state using armed force (in a time-honoured tradition, going back at least to the Greek tyrants starting with Cypselus) but it's much more difficult to control a large state using mercenary troops - one would only be in possession of the capital and even if the rest of the country doesn't rise up against the usurper, how would one collect the revenue? And without revenue, you can't control your mercenaries.
This issue was grasped well by the contemporaries. The only example that comes to mind of a mercenary captain who tried to turn himself into a ruler is, of course, Wallenstein. And even in his case, it is important to note that before he could really proceed with his ambitious plans he first had to acquire a solid territorial base, the duchy of Mecklenburg, in the "usual way" - receiving it from the Emperor.
UPDATE: So far I've explained why a coup by a mercenary captain stood no chance of success. Regarding the wider issue broached at the title of the question: why wasn't the ruling class subverted by the mercenary captains, the answer is actually quite simple: the captains belonged to the ruling class themselves.
Upvote:0
The model for "mercenary" armies in say, the 17th century, was that of Albrecht Wallenstein, not that of say, Johann Tilly.
Although he had and used mercenary troops, Tilly was a member of the "Establishment" who also had, and used "Imperial" (read "establishment") troops. On the other hand, Wallenstein was a "random" noble out of "nowhere," who raised mercenary troops on his own, initially for the Catholic side in the 30 years' War.
Wallenstein managed to defeat lesser Protestant commanders such as Mansfield in Moravia, but had a mixed record against Sweden's Gustav Adolphus, before the latter's death. At this point, Wallenstein decided to favor "German" rather than "Imperial" interests, and try to negotiate peace with the Protestants. These people distrusted him, and the Emperor considered it treason, and "proscribed" him.
Basically, "pure" mercenaries like Wallenstein were eventually considered "too hot to handle" and their powers were limited by their "handlers" and opponents. Others, like Tilly, who used "some" mercenary troops, were bona fide members of the ruling class, and no threat to it.
Upvote:7
I'd like to expand Felix's answer in relation to soft-power (such as "legitimacy"). Mercenary captains did not participate in the reproduction of feudal wealth, they merely directly expropriated it (or received payment from the emergent state). Traditional rulers participated in the full reproduction of economic life, they imposed status on the third estate and benefitted from taxation and corvee. Mercenaries were not imposing such relationships on the level of the traditional domains controlled by the traditional ruling class. Mercenaries ravaged crops, they didn't keep peasants infeudated such that they would prepare a crop for the next year.
So there wasn't a serious threat to the structure of the existing ruling class because Mercenary captains (unlike the burghers of the cities), did not offer or suggest a different social organisation of productive relationships.
As far as the actual composition of the ruling class changing, Felix admirably uses Wallenstein as an example. There's no point in killing off the existing ruling class, and substituting yourself for them, unless you can organise the hegemonisation of power. And here soft power ("legitimacy") is important. If you look at early modern disputes over legitimacy, consider the sordid business of England, an illegitimate ruler faces an extended multigenerational period of disquiet from minor members of the ruling class. Easier to gain access yourself to the existing system of the ruling class, than to disrupt it. A smaller slice of a bigger pie, rather than a whole small pie. In fact, as we can see in the emergence of the Absolute State in France, the existing ruling class actually preferred a stronger and stronger capacity for force by the Crown as an abstract principle in this period, the better to enforce higher rates of feudal extractions.