score:4
In general, the longstanding legal precedent in the USA is that you cannot have your citizenship involuntarily revoked, and Afroyim is really a re-establishment of that precedent, not a deviation from it.
The "precedent" you are probably referring to is most likely Perez v. Brownell. It was at the time (to my knowledge) the only Supreme Court ruling ever explicitly upholding the right of the State to involuntarily remove someone's US Citizenship*. Acting against that case were:
What happened in the meantime is what always happens with unusual 5-4 decisions: As soon as the membership in the court changes, lawyers start to test the decision by bringing up cases against it. By 1967 by my count the court had 3 personnel changes since that decision. By the time Perez' final test came with Afroyim, it was found wanting.
* - This point's arguable, indeed that's why we still need courts. :-)