How did the Mongol Nomads' System of Raiding and Trading Work?

I am not an expert on the Mongols, but what you describe seems similar to the Irish Tain Bo Cualinge, or the Norse Viking around the period of the Great Heathen Army (that's not the best reference, but it is the one I could find quickly - I'm thinking more of Rollo, and possibly of the Normans in Sicily (I think there is a podcast by Lars Brownnworth, but I can't remember)

Essentially in premodern societies, relations with external groups were very fluid; depending on relative wealth, prosperity and population, they could be partners in trade, or targets of raid. This was normal behavior.

OP insightfully points out that this behavior is "pre-nationalism". All of the cited societies are not "nations". That observation is completely in line with my reading of Fukuyama. Not my field of study, but it seems to me that one of the characteristics of Nationalism is the establishment of a "foreign policy". A stronger sense of internal identity facilitates a desire for consistent and coherent external relations. Not sure how I'd test that theory, but I'm confident that H:SE will point out what I've missed.

My personal theory is that we're seeing a consequence of shifting internal factions in a society that is at the limit of the ability of the political institutions to create and manage a group, but I have no evidence for that.

Yes, there is (was?) such a custom in the Eurasian steppes (i.e. Mongols, focus of the question). I don't know it terribly well because this one is certainly within social anthropology+, not really my interest and reading. But, I'm going to try to explain it.

1. Selective raiding and pillaging between tribes is -- within their context -- a practise that seems quite common and accepted. This type of inter-tribe raids does not involve violence (i.e. they steal and run-off).

2. These raids, on the whole, are considered acceptable, because, to them, being opportunistic by using guile, agility, etc is a badge of honour. A "nomadic code of honour & bravery" to show they dare to and am able to do it, if you like.

3. There are even classifications of raiding their neighbours, with specific Turko-Mongol names -- training raids, revenge raids,etc. I cannot remember the Turkic names .

4. Finally, these 'cattle stealing raids' between themselves should be differentiated with the bigger 'tribal conquest' type of raids. For that, death and violence is the norm because they are going all-out -- against sedentary societies and other unfriendly tribes -- what they are really going for is booty (ölgä in Turko-Mongolic) and it is important to do so because the redistribution of booty is one of their form of leadership and gaining reputation/support (let's call them war-band raids).

As Mark said earlier: " a consequence of shifting internal factions in a society that is at the limit of the ability of the political institutions to create and manage a group", this is about leadership and rewarding the team (i.e. statehood and governance).

Social Anthropology+:This topic is normally beyond me, so I have to go find details and source. If you are looking for a book about socio-economic systems of nomadic pastoralism culture, I'd recommend Professor Anatoly Khazanov's book,"Nomads and the Outside World" (University of Wisconsin Press; 2 edition, 1994). Prof Khazanov formalised the extended definition of nomadic pastorlism (still in use).

So, to recap and end, there is a Mongol custom and classification that is highly nuanced when it comes to raiding - i.e. cattle raids & war-band raids. Because of this, they do not have to kill-off (exterminate) everyone of the losing side -- much depends on the context.

The immediate example I can think of is, even during the dynastic fights between Jochid and Toulid ulus, Berke and Hulegu were not at full-massacre mode (i.e. not killing-off each other, it was a minor-raid), as these passages show:

Under the command, probably nominal, of Abagha, Hulegu's son and future successor, the Ilkhanid force advanced into the Steppe, crossed the Terek River, and came upon Berke's deserted but well stocked winter encampment (qishlaq). For three days the Ilkhanid troops indulged in merriment, until they were surprised by Berke's forces and completely routed (1 Rabf 1661/14 January 1263). Retreating across the frozen Terek, Abagha's forces suffered another disaster, when the ice broke under their weight and many troops were drowned (painting by Hayton of Corycus). Abagha himself escaped, and his surviving soldiers were pursued to the southern end of the Darband by Berke, who then returned to his own country....

Of great interest are Berke's words, as reported by Ibn Wasil and later sources, upon surveying the carnage on the battlefield after Hiilegu's army had been defeated. Bemoaning the large number of Mongol dead, he cursed Hulegu and said: "Mongols are killed by Mongol swords. If we were united, then we would have conquered all of the world."

SOURCE: Reuven Amitai-Preiss, 'Mongols and Mamluks: The Mamluk-Ilkhanid War, 1260–1281' (Cambridge, 1995), pp.79-80.

The fact that Abaqha and troops could 'return back' tells us that Berke's army didn't finish them off. Remember, Jebe and Subotei chased the Shah (Muhammad II) to the Caspian Sea during the Khwarezm campaign. (I placed that additional paragraph there because of that famous quote -- just wanted to show the context of that quote).