How formalised were the various tiers of Russian nobility during the regency of Sophia Alekseyevna?

17th Century Russian Nobility

There were indeed numerous ranks within the nobility:

1. boyars
2. okol'nichie
3. dumnye dvoriane

In general, Wikipedia's Russian nobility is instructive though it focusses on the 19th century. However, it does describe circumstances where nobility (formalized status of nobility, that is) could be granted, such as by Imperial grant or by reaching a certain rank in the military.

Upward mobility was curtailed by one's relations: for example, someone couldn't be made a boyar unless they already had a relation who was at least an okol'nichie (from one of the non-Wiki articles I've linked to, but I lost the specific one).

A characterization of the dumnye dvoriane before the 17th century is useful, indicating a far looser historic understanding:

Syn boiarskii (p1. deti boiarskie) refers to the lesser gentry in the Muscovite state, who provided the majority of the tsar's military servitors in the sixteenth and seventeenth centuries. ... According to Richard Hellie, “The term literally means “boyar children,’ which may mean either that initially they were the sons of boyars, or, more likely, simply the retainers of boyars. By the mid-sixteenth century, at the latest, the term had neither of these meanings, rather signifying simply landholding members (pomeshchiki) of the cavalry. In the sixteenth century deti boiarskie were recruited from all social milieus, including cossacks, peasants, and even slaves. This avenue of social mobility was closed at the beginning of the seventeenth century as the rank became hereditary.”
_{—Kleimola, 'The Duty to Denounce in Muscovite Russia'}

Therefore, we recognise that there was fluidity within the ranks between the 16th and 18th centuries. The abstract for another article confirms as much with respect to the upper ranks:

The later seventeenth century was an era of unprecedented social mobility in the upper reaches of Muscovite society. Prior to the reign of Aleksei Mikhailovich, the boyar duma had been the preserve of a small set of pedigreed families. Aleksei, however, altered the traditional duma recruitment policy in the 1650s and began to promote undistinguished "new men" into the duma. Despite the claims of some historians, the new men were not radicals. It is true that many of them had made their way to the top by virtue of their service and skill, and not due to any hereditary right to elite ranks or offices.
_{—Poe, 'The Imaginary World of Semen Koltovskii'}

The same article also indicates that the dignity of one's family would indicate status in the Muscovite court, 'the status of which was measured by its antiquity and the quality of its service to the grand prince' and because of that, many families tried to 'fabricate' longer histories for themselves.

Crummey's 'Aristocrats and Servitors' looks to be an useful reference work if this topic is of further interest.

The Title of Kniaz

Wikipedia's kniaz explains the title:

Kniaz continued as a hereditary title of Russian nobility patrilineally descended from Rurik (e.g., Belozersky, Belosselsky-Belozersky, Repnin, Gorchakov) or Gediminas (e.g., Galitzine, Troubetzkoy). Members of Rurikid or Gedyminid families were called princes when they ruled tiny quasi-sovereign medieval principalities. After their demesnes were absorbed by Muscovy, they settled at the Moscow court and were authorised to continue with their princely titles.

Prince Vasily Golitsyn

In Prince Vasily Golitsyn's case, he was descended from Gediminas of Lithuania (either thirteenth or fourteenth generation), hence his use of the princely title. He was, at the same time, a member of the boyar grouping, the two not being connected to each other.

I found a book, by Lindsey A. J. Hughes, "Russia and the West, The Life of a Seventeenth-century Westernizer, Prince Vasily Vasil'evich Golitsyn (1643-1714)", which sounds as if it would answer your case in detail. I didn't find an online copy.

However, a review of the book mentions that Golitsyn died in exile, but not his status at death—though the review naturally covers only a fraction of the length of the book. Hence, the revocation of Golitsyn's status cannot be discounted. De Madariaga's brief biography of Dmitry Golitsyn, a cousing to Vasily, also mentions only exile.

Summary

With this in mind, but without access to the actual biography of Vasily Golitsyn, I would guess that either the tsar had the power to 'cast someone down' (though I couldn't find a mention of this), or that downward mobility was possible along the same lines as upward mobility (i.e., within the ranks of nobility), or, indeed, that there has been a misunderstanding of Vasily Golitsyn's specific rank at death.