Inquiry into the nature and roots of the temporal power of the Roman Catholic Pope:

Around the turn of the 15th and 16th centuries, Popes such as Alexander VI and Julius II greatly expanded the territory, and resulting secular power of the Papal States.(The territory controlled by the Popes then encompassed most of north-central Italy except for Tuscany.) Coincidentally or otherwise, this coincided with the decline of Popes' temporal power. The Popes came under attacks from Protestant reformers (that would soon include Martin Luther and Henry VIII), displeased with their religious and personal activities. But these "activities" consisted largely of using their new-found secular authority to help maintain the balance of power in Italy, and hence in Europe. It is because of this balancing act, as much as religion, that gave the Popes their political power after the Middle Ages, in light of the secularization of royal power in Europe.

Fast forward to the 20th century. In 1929, after the 19th century unification of Italy, the Italian government and the Holy See signed the Lateran Accord that defined the Pope's temporal power over the Vatican. This ratified the formal sovereignty ("monarchy") granted to the Pope in the original 1870 liberation of Rome, and gave the Pope sweeping powers over his domain.

The real key to the Pope becoming a temporal lord was not the Fall of the Western Empire, but later. When Odoacer became King in Italy after 476, and Theodoric the Ostrogoth after him, the Popes were normal (if influential) clerics and with land in and around Italy and Sicily. These rulers were Aryan Heretics, so the Orthodox Popes were restive but still not temporal leaders.

Then came Justinian's reconquest and the 20 year Gothic Wars. The Popes now were subjects of the Eastern Empire yet still were just normal clerics, if influential. There was a certain amount of turmoil between the ideas of East and West, with Popes being arrested and carried off to Constantinople from time to time.

The big change came with the Lombard Invasion around 568. The Imperial hold on Italy was reduced to pockets here and there, with the governor in Ravenna and the Popes in Rome and Lombard states between. The Lombards themselves broke up into small dutchies in the highlands in the middle of the country.

So the Pope was faced with a threat - the Lombards might take Rome - and no secular support from any larger state available to help out. The Popes were able to play off their religious influence to keep the Lombards at bay and become the defacto temporal lords. They had a run of strong-willed and competent Popes, which helped.

After that, the Popes seem to have as their policy to try and avoid being subjects of a powerful lord. Thus the "Donation" fraud to have Charlemagne stop in Northern Italy, and the later conflicts to limit the powers of the Holy Roman Emperors in Italy.

As you rightly say, the history of the pope's temporal power is sure to be complicated, but I would think the following points should be noteworthy as an overview.

To begin with, it is probably reasonable to assume that the Pope held non-trivial worldly authority ever since the Emperor Theodosius made the Christian church the state religion of the Roman Empire. In ancient times, there was generally far less division between spiritual and temporal powers. The title of Pontifex Maximus is noteworthy in this context, seeing as how it was claimed by the Emperors themselves and therefore coincides with the political head from the Principate era onwards, and was later also claimed by the Popes. That alone seems to indicate some connotation of worldly involvement. Further intercourse between the Popes and the emperors can be seen by the creation of the permanent office of the papal apocrisiaries. I would dare speculate that the Pope holding temporal power was probably a less controversial issue in the early church than it would have been since and during middle ages, when there would have been a more contentious balance of temporal power between the Holy Roman Emperor and the Roman Pontiff.

In the specific case of the Pope holding true monarchic powers, however, I think it's safe to say that they were definitely established with the donation of Pepin in 756 of what became known as the Papal States to Pope Stephen II in return for Stephen strengthening Pepin's authority as King of the Franks by anointing him (which in turn was in return for Pepin's support against the Lombards). However, to further drive the point that the Pope's temporal authority was not an innovation by this donation, it should be noted how previous Popes held some worldly control of Rome itself and fluctuating territories outside of Rome. For instance, Gregory III oversaw the reconstruction of the Walls of Rome as against the Lombards. Gregory II used his authority to drive the governor of the Emperor (by that time, seated in Constantinople) in Rome from the city. As early as the sack of Rome in 410, Innocent I was one of the envoys sent to Ravenna by the Senate, indicating his involvement in worldly affairs.

The Vatican is just what is left of the Papal States since everything else was taken by the Carbonari during the Risorgimento. It is noteworthy, in this context, that the Popes did not formally accept that situation for more than half a century (after which the Vatican was recognized as a sovereign principality), considering themselves prisoners in the Vatican. By this time, the idea of the Pope wielding temporal powers had become so entrenched that, as Wikipedia puts it, "they claimed that total sovereignty was needed so that a civil government would never attempt to interfere in the governance of the universal Roman Church."