You will go through security the moment you land (which will be in Terminal 3, if I remember FCO correctly).
You are entering the Schengen zone in Rome. That's a passport control.
Regarding customs, EU differs from the USA and Canada: you do not need to pick up your baggage if you are on a single ticket. You pick up your baggage at the end (Venice) and go through customs. If you have multiple tickets the situation is murkier: you might or might not be able to get the agent in Boston to check your luggage through (if the two airlines have interline agreements or if it is a single airline). This will be very clearly communicated to you and also you will get baggage slips.