Do I need to carry my passport while travelling within the Schengen area?

For EU citizens it might not be strictly necessary to carry an ID when traveling around the Schengen area but you have to make several distinctions here. The fact that the police can and does check ID does not mean it’s mandatory for everybody to have one on them. Conversely, the fact that there should be no systematic check at the border does not mean that it cannot be mandatory to carry ID (it’s the case in the Netherlands for example and yet actual checks seem rare in my experience). Rules can also be different for locals, EU citizens and third-country nationals.

For example, regarding the first incident, French citizens are not even required to hold a national ID card at all but it’s still perfectly legal for the police to do ID checks under certain conditions, even far away from the border (more details here or on service-public.fr). I think that even for third-country nationals (and certainly for French citizens), not having any means to prove your identity on you is not per se a punishable offense. The police might keep you for a few hours to ascertain who you are by other means or be generally unpleasant but you cannot be charged with anything just because you left your passport at home.

Furthermore it should be pointed out that racial profiling has long been a problem in France. I have no way to independently assess how widespread it really is but I heard many anecdotes and some organizations regularly complain about it. There are even a few official reports that acknowledged the problem and, back when he was still running for office, the current president promised a number of specific measures to deal with it. The details are complex but this gives you some idea of the context.

Now, under the Schengen agreement, France committed herself not to perform systematic checks at her borders. But anything else still depends on national law. For example, being part of the Schengen area does not mean that nobody can ever be asked for ID on French territory. Nor does it provide you with any practical recourse if you are victim of racial profiling or if you suspect French police is bending the rules. If France really is in breach of the Schengen agreement, this is something other EU member states and especially the EU Commission might deal with in several ways (potentially escalating to an infringement procedure in front of the EU court of justice) but that does not immediately help you in practice.

In fact, French law even specifically allows the police to ask people for ID at all times in border areas, on international trains and in a number of train stations including Metz. In 2011, the Commission looked at what France was doing and issued a critical-sounding press-release but did not go further or challenge the law itself. This guy has been writing extensively about checks at the France-Italy border and elsewhere so it seems to be a relatively common occurrence.

All this also means that there are no EU or Schengen-wide rule that would definitively answer your question. It’s certainly not mandatory to always carry ID everywhere but since police checks do happen, carrying your passport seems like a good idea in practice, especially as a non-EU citizen.

By law most EU nations require all people to carry photo ID at all times and present it to police when requested.
There’s usually a hefty fine for failure to be able to do so. And possibly a trip to a police station for fingerprinting and other means of trying to find out who you are.

While it’s not common for police to be around, even when you need them, and even less common for them to ask for your ID (unless maybe you get caught up in a riot or major search operation, things like that), you can be asked at any time so best be prepared.
Mind that that doesn’t have to be a passport, a European identity card or driver’s license is usually enough (but might not be for non-EU citizens).

I was asked for my passport on the Metz, France train station while leaving for Luxembourg. Note: This was NOT ON the train but simply on the station because I was clearly a foreigner.

This was an anomaly, I would believe. I have moved around quite a bit in Europe. Always carry my passport with me, but not for this specific reason, and have never been asked to take it out in the past 5 years even when I have crossed borders within the Schengen region. Don’t play the foreigner card(alibi). I am also a foreigner, and I have visited places where it is next to impossible to find a coloured person for miles in either direction.

Anyway when you applied for a Schengen visa, there should have been a printed note returned with your passport which says you have to carry your passport, proof of funds etc around always. Didn’t you get one ?