How were "Wehrbauern" (soldier peasants) in medieval Europe even a thing, at all?

Let us take the example of the empire of Charlemagne and its expansion over the Pyrenees, in the late eighth century, clashing with the Umayyad Caliphate which controlled the Iberian peninsula. This is a real situation from which we can see patterns that recur with other polities, and other borders. The Wehrbauern, later and elsewhere, actually come from the same post-Roman Visigothic system of land tenure and obligation that was behind the structure used here.

The Franks were a large and powerful kingdom. They had a King, with a roaming court rather than a fixed capital (Charlemagne used Aachen as a home base but it was not an administrative headquarters independently of his presence). He had a core group of elite troops, the scara, but these were not enough to constitute a full army. To get more people, he had to summon them, according to a set regime. This levy obliged some local authority, such as a count or a bishop, to provide men; there were rules about repeated service, penalties for non-performance, and so on. Ultimately, that reflects a negotiation between competing demands, including the need to have people working the fields. Charlemagne's practice was to have a summer campaign each year.

Some of those offensives were targeted at creating a barrier region south of the Pyrenees. While the mountains are a natural boundary, control of the southern side would prevent the Umayyads from opportunistic raids, and the land was desirable in itself. But that territory, the "Marca Hispanica", was also then exposed to attack from the remainder of al-Andalus.

Charlemagne established the usual pattern of provincial rulers, in this case known as marquises for those territories on the border. "Marca", "marquis" and "march" are all from a Germanic root meaning "border". These rulers were not yet hereditary. They were in charge of their area, with considerable autonomy since at most times, Charlemagne was far away. While subject to the usual annual levy, they were also expected to be on the defensive at ordinary times, so that the whole area could remain Carolingian.

There was no way to support an additional standing army based on the resources available, and within the constraints of the system.

• Levied soldiers are obliged to return home. They have to help with the harvest. While this point might be pressed, even Charlemagne has limited ability to anger his vassals.
• All soldiers have to be housed and fed. The mobile campaign had strict logistics to make it possible for the army to survive. These were not available in the case of longer-term occupation. The population was initially quite low in the conquered regions.

The counts and marquises did use their own scarae, but we are not talking about huge numbers of people: they also relied on the ordinary populace at need. Specific forms of local levy were used to create castle garrisons, later known as the gaitagium or watch-duty, and counts would have even-more-local subordinates. In Catalonia, the smallest unit was the villa, a grouping of a half-dozen peasant families who paid their taxes to the count as a community.

Desiring to expand the population (which would increase production and make a larger body available for the levy), Charlemagne offered increased autonomy in some of these locations, under a system of land-grant called the aprisio. Incoming landowners would take charge of some swathe of previously unoccupied land, and after thirty years would gain legal tenure almost amounting to allodial title - a high degree of security, subject to the payment of taxes and participation in the levy. They and their descendants became proprietors of their own land, independent of the everyday control of the local lord. This was an attractive offer, even if it came with the understanding that they would occasionally be attacked by the Umayyads.

While Charlemagne and successors were giving up a degree of control, holders of land per aprisionem were still integrated into the political, economic, judicial and cultural system of the empire. Local lords were the ones losing out, which did give rise to various armed clashes and legal changes over the next centuries.

To follow up on some specific points raised:

• "Essentially free roaming bandits in the border regions" is not accurate; the point of peasants is that they were settled on the land. Like anybody else, they wouldn't be off fighting willy-nilly, but only when there was a real need. They were not conducting bandit raids on their neighbours in the March.
• "reducing your leverage over them" - ultimately, you are still getting what you care about (taxes, levies), and maybe even achieving more loyalty and prosperity
• "cast off the yoke of their feudal serfdom" - this is an important dynamic, but often taking the form of incremental changes to conditions rather than mass armed rebellions against the aristocracy. There were many clashes between and among peasants and lords, but not framed in terms of class-consciousness as such. It was more often about specific grievances.

Similar patterns have been used at other places and times in history. That includes the Roman system of emphyteusis and the American settlement of its western frontier, which are both cases where there was a strong but distant central administration. Local autonomy was going to be the case anyway, but can be formally offered in order to entice people to dangerous duties.