How widespread were major dynasties which did not follow agnatic succession or male primogeniture?

DVK's question is flawed. Agnatic means through the male line of descent. The present Swedish and British law is called absolute primogeniture, with the eldest child succeeding whether male or female. There is nothing agnatic about absolute primogeniture and it is not some subset of agnatic inheritance. Thus DVK's terminology is flawed.

There have been many different customs and rules of royal inheritance, usually similar to or even identical with the customs and rules of property inheritance, through the ages, and those rules have usually been made into law in recent decades and centuries.

One large category is primogeniture, in which the first born inherits.

1) Absolute primogeniture is the current rule for royal succession in most of the remaining monarchies in Europe and in the Spanish nobility. The first such law was passed in Sweden in 1980, and by now the only European exceptions seem to be Spain, Monaco, and Lichtenstein. In primogeniture the children and other descendants of the previous monarch have priority over the previous monarch's siblings, uncles, aunts, nephews, nieces cousins, etc.

In absolute primogeniture the oldest child of the monarch, whether male or female, inherits the throne, or lacking an oldest child, the closest thing to the oldest child of the oldest child, etc.

2) In agnatic primogeniture or patrilineal primogeniture, the inheritance can only go through and to males. Thus brothers, uncles, nephews, cousins, etc. of a monarch would inherit instead of his daughters, female children of his sons, or even male children of his daughters. There have been famous examples of distant male lineage relatives inheriting under this rule when many other persons were more closely related to the dead ruler.

When King Henry III of France Was assassinated in 1589, King Henry III of Navarre succeeded as King Henry IV of France - though he had to fight a civil war to do so. Henry IV was related to Henry III in the 23rd degree. If you counted Henry III's and Henry IV's ancestors back to their common male ancestor King Louis IX (died 1270) the total was 23. Louis IX died 319 years before Henry IV inherited.

At that time King Henry III had a surviving sister - the childless wife of Henry IV - and his heir by male preference primogeniture was Spanish Infanta Isabella Clara Eugenia.

When Elector and Duke Maximilian III Joseph of Bavaria died in 1777, his closest relative was his sister Maria Antonia Walpurgis (1724-1780), Electress of Saxony. But because of agnatic primogeniture his distant relative Karl Theodore (1724-1799), Elector and Count Palatine of the Rhine, inherited Bavaria. Their latest common agnatic ancestor was Duke Louis II (1229-1294) the Severe of Bavaria, who died 483 years before Karl theodore succeeded to Bavaria.

When William III of Nassau (1817-1890) King of the Netherlands and Grand Duke of Luxemburg, died in 1890, his daughter Wilhelmina (1880-1962) became Queen of the Netherlands. But the succession law in the Grand duchy of Luxemburg was agnatic, so his distant cousin Adolphe (1817-1905) Duke of Nassau from 1839-1866, became Grand Duke of Luxemburg. Adolphe was his 17th cousin once removed, allegedly the most distant relative to ever inherit a throne. Their latest common agnatic ancestor was Count Henry II (c. 1190-1251) the Rich of Nassau, who died 639 years before Adolphe inherited.

3) Male-reference primogeniture or cognatic primogeniture was probably the most common succession rule in the middle ages. That meant that a ruler's descendants were preferred to those of his siblings or more distant relatives and that brothers were preferred to sisters. But if a ruler had no surviving sons or sons of his sons, his daughters, or his son's daughters, or his daughter's children, would be preferred to his siblings or more distant relatives, even the male ones.

This became the most common rule of succession in Europe in the Middle Ages and Modern Era. In England the first two Queen Regnants were Jane and Mary I in 1553, but as early as 1135 the two rival claimants to the English throne were King Henry I's daughter Matilda and his sister Adela's son Stephen. Richard Duke of York claimed the English throne in 1460 by descent through two Women.

In Scotland in 1284 King Alexander III (1241-1286) made the nobles swear to recognize his granddaughter Margaret (1283-1290) the Maid of Norway (daughter of Queen Margaret [1261-1283] of Norway) his heir if he had no other children in the future. And after Margaret died all the many competitors for the Scottish crown except one claimed it through one or more women.

By coincidence, 19th century Spain and Portugal both had succession wars between uncles claiming agnatic primogeniture and their nieces claiming male-preference primogeniture.

4) Uterine primogeniture. Under this system the oldest son of the nearest female relative inherits. Thus a monarch would usually be succeeded by his oldest sister's son. It has been claimed that the Picts in early medieval Scotland practiced that form of royal succession. Some tribes and kingdoms in Africa follow uterine primogeniture.

5) matrilineal primogeniture or female-preference uterine primogeniture. This is the mirror image of agnatic primogeniture. The throne is suppose to pass from female to female, to the oldest daughter of the oldest daughter of the oldest daughter. The Modjadi or Rain Queen of the Balobedu people in South Africa has that succession law.

6) I guess the opposite of male-preference primogeniture would be female-preference primogeniture when the oldest daughter of a queen would normally inherit, but a male could inherit if he didn't have any sisters.

Non primogeniture succession.

There were plenty of succession rules, customs, and laws that did not involve primogeniture.

7) Agnatic seniority is the system used by Saudi Arabia. the oldest male in the family is usually the next heir. Thus all the Saudi Arabian kings for 65 years since 1952 have been brothers and half brothers, sons of the founding king ibn Saud. Most of the kings had adult sons of their own that were passed over in favor of their older uncles.

8) the Rota system is the modern name for the succession rule in Kievan Rus. There were a number of principalities ruled by princes under the Grand Prince. When the grand Prince of Kiev died some other prince had to move to Kiev to become Grand Prince and there was a reshuffling of principalities among surviving princes, presumably in the order of seniority. There were of course a number of civil wars fought about succession.

The Princes modified the rota system in a conference in 1097. The princes got to keep some lands as patrimonial lands outside the rota system. One patrimonial land, the principality of Moscow, was the core of the Moskovite state and of modern Russia, expanding as the rota system gradually faded away.

A similar system was used in Poland from 1138, and in Movavia and Bohemia from 1055 to 1182 and 1203.

9) The royal succession method in the Gaelic lands of Ireland and Scotland was complicated. Membership of the dynasty was limited to male lineage males, as in agnatic primogeniture. But instead of hereditary succession from father to son, every adult male closely enough related to the king was a potential successor. The group of dynasts was often limited to the adult sons, grandsons, and great grandsons of a previous king. The eligible dynasts elected a Tanist as the heir to the throne. When the king died the tanist succeeded him and the dynasts elected a new tanist. Sometimes a king's son was the tanist, sometimes a brother, or nephew, or uncle, or first cousin, or remoter cousin.

The tanist system enduring in Ireland for about a thousand years from the dawn of Irish history to the 16th century when Irish kings surrendered their kingdoms to the English Kingdom of Ireland and received them back as noble fiefs descending by primogeniture.

King Malcolm III of Scotland (reigned 1005-1034 challenged the tanist system by choosing his maternal grandson Duncan I (c. 1001-1040) as his tanist, and the descendants of Duncan's son Malcolm III (c.1031-1093) managed to make the throne hereditary by male-preference primogeniture.

10) Welsh land inheritance rules were that land was divided equally among all the sons, legitimate or illegitimate, of the deceased. Welsh royal inheritance involved one son inheriting the kingdom and his brothers usually inheriting regions to rule as sub kings or lords under his authority. A Welsh king was supposed to be the son of a previous king but not necessarily the immediate previous king, thus kings were usually sons, or brothers, uncles, nephews, or cousins of the immediate previous king.

11) Patrilineal ultimogeniture. Inheritance by the youngest son.

Patrilineal ultimogeniture, where the youngest son inherits, was customary among a number of cultures including: Fur, Fali, Sami (also called Lapp), Bashkir, Chuvash, Gagauz, Vep, Tatar, Achang,Ayi, Atayal, Kachi, Biate, Chinantec, Hmar, Mro, Kom, Purum and Lushei or Lushai (sometimes mistakenly taken for the whole Mizo people, especially in the past).

https://en.wikipedia.org/wiki/Historical_inheritance_systems[1]

And there have been many other inheritance customs and rules and many variations on the main systems.

The Inca chose their ruler from among the sons of the prior Inca. A group of the prior Inca's advisers, generally made up of priests but sometimes military advisers as well, selected the most qualified of the legitimate sons for the post. Because they were unable to accept any cultural norm but primogeniture for deciding inheritance the invading Spanish labeled most of the historical Incas as 'usurpers'Commentarios Reales, Garcilaso de la Vega.

Unfortunately the lack of a formal system for nomination of the new Inca terminally weakened the Empire when the military advisers in Quito selected Atahualpa to succeed Huayna Capac, while the religious leaders in Cusco selected Huascar.