When travelling do I have to keep the passport with me all the time?

I have been really fortunate to travel all across the world. My rule of thumb is: don't let your passport out of your sight. When anyone smarmy walks away with your only legal means of exeunt, that's when the shakedowns and baksheesh starts to happen. Instead, leave your passport in a safe place and travel without it, but with a photocopy and a very official looking stamp. Preferably one that says "Certified Copy" in a few languages. Also, get yourself a fake International Driver's License for 10 dollars in Bangkok. Worth its weight in gold.

Assumptions

For the purpose of this questions I am assuming that we are talking about a traveller who is legally allowed to be in the country at hand, and therefore is more than willing to cooperate with the officers performing the ID check.

ID in Italy

Italy is a country which issues national identity cards, very much like France and Netherlands. Carrying some form of identification isn't mandatory in Italy. However, if questioned, one is obliged to give Police details about one's identity. In addition the Police can take you to the station for further checks in case you fail to convince them. You will be held in custody until they have resolved the issue. Obviously, carrying a valid form of ID on you at all times helps avoiding this.

If you ever were to travel to Italy, I would leave the passport in a safe in the hotel/hostel. Worst case scenario you would explain your situation and be escorted to your accommodation to pick-up the passport. If you want to be extra safe you can carry a photocopy of your passport (and visa page, should you require one). I would say this is strategy which applies to all countries.

Finally, since for Italian citizen, the driving licence counts as a form of ID, if worst comes to worse you could always show that to the police officers and explain your situation.

It depends on the law of the countries and the specifics of the situation. Personally, I only know one country where carrying ID at all times is mandatory and failing to have one on your person can result in a fine (it's the Netherlands) but there are certainly others. Wikipedia has a overview of the requirements in many countries. It's mostly about the documents locals (and not visitors) must carry but as a first approximation, I would assume that if nationals have to have ID at all times, foreigners are also expected to carry a passport.

I suppose there are many countries where the police can make your life miserable (e.g. bringing you back to the police station and keeping you for several hours) but not fine you or punish you merely for not carrying ID (assuming you are legally in the country and do have an ID somewhere, only not on you). I know it's the case in France for example. Similarly, in Germany, it's mandatory to have ID but not to carry it with you (I don't know exactly what the police can do if you don't).

And then the letter of the law is only part of the problem. If some official happens to insist on seeing a real passport, arguing about legal details, in a foreign country, possibly in a language you don't speak well, might not be the best way to stay out of trouble.

Therein lies the problem with strategies like carrying a copy of your passport. It can be useful and will probably be enough in many situation but legally speaking it's probably not different from having no ID at all and you most likely don't know whether it is and are not in a good position to defend your rights. So in practice a copy of your passport might not be much help against a policeman bent on making trouble or a corrupt cop trying to extract a bribe from you.

But in the EU at least, if you are an EU citizen, carrying a national ID card should be enough for all purposes (if you have one obviously, some countries don't issue them at all). So you can indeed keep either your national ID card or your passport safe, use the other one as needed and if it gets stolen you can easily return home and renew the stolen ID using the one you stashed away.

You don't state your nationality, but I was looking through the Foreign Travel Advice section of gov.uk and found two examples:

The Equatorial Guinea page states:

Be alert and take sensible personal security precautions. Roadblocks and unannounced identification checks are likely. Carry an appropriate form of identification (passport or residence permit) with you at all times. See Crime and Road travel.

Then, for instance, the Zambia travel advice, crime section states:

Keep valuables and originals of important documents in a safe place and carry a copy of your passport and immigration permit.

This tells me that each country would have its own advice and they won't all be the same, so, as Gayot Fow states in his comment, it would be not possible to generalize.