Upvote:8
Actually, not that many countries require a visa for US citizens and even among the most restrictive/unfriendly countries, only a handful would have the resources to track and identify random low-level intelligence staff so I would not expect a huge impact.
On the other hand, some countries (and especially the US) place restrictions on foreign travel by clearance holders. So if you want to keep your job and your career, you need to follow specific procedures and cannot go abroad just like that.