Upvote:5
The rules are explained on the official EU website. It sometimes difficult to understand how they apply to a given situation but the page should be authoritative. In your case, my understanding is that you do need a visa to visit other European countries.
To understand why, you have to make a few distinctions. Assuming you are talking about a short stay, there is no such thing as a “EU visa” but several distinct area/countries with their own visa regime:
Croatia, Romania, and Bulgaria have rules modeled after the Schengen area and make it easy to visit for holders of multiple-entry Schengen visas or Schengen residence permits but are still separate. UK and Ireland have entirely separate policies and don't have any facilities for Schengen visa holders but do have to honor a few EU rules.
There are two ways in which being the spouse of an EU citizen can allow you travel to some or all of these countries without a visa but neither apply to you as far as I can tell:
In all other cases (and in particular if you live in your spouse's country of origin), you have to follow national rules regarding spouse visa (i.e. you don't get an “EU family member” permit) and you do need a visa to travel to other EU countries. The good news is that as long as you travel with your spouse, this visa should in principle be issued quickly and free of charge.
Note than in the UK, British citizens have successfully argued that staying elsewhere in the EU for some time is enough to invoke EU rules instead of national rules (it's called the Surinder Singh route after the person who argued for it in front the EU Court of Justice). In that case, you would initially have applied for an EEA family permit instead of a family of settled person visa and you would fall under the second exemption I mentioned above.