tl;dr In lack of a valid ID, take your passport with you.

There are a few misconceptions in other answers or comments (though Tor-Einar Jarnbjo got it basically right).

You are required to own a valid ID (or passport and Visa, and other things, it depends) to enter Germany as a foreigner. You are also required to own a valid ID, alternatively a passport if you are German and over 16 years of age (and not infirm and permanently hospitalized). Mind the wording, valid.

A (valid) passport qualifies as valid ID, a driver’s license (or anything else, including your library card) does not. Though some non-officials may accept a driver’s license anyway.

Note that “valid” also applies to expired or expiring IDs. Be sure your passport doesn’t expire. Failure to possess a valid ID is subject to a fee up to 3,000 Euros, and it is up to the officer to decide whether and what amount of fine you pay (nope, that’s no joke). I’ve once paid 200 Euros because my ID expired two days before my being-renewed ID was ready, and my passport had expired some weeks earlier, too.
Which is just kafkaesque because the officer only noticed my old ID had expired two days earlier when I went to get my new, valid, ID. Which had been ordered some weeks earlier showing a then-valid ID, and which was, of course, perfectly valid now, and there was never at any time any doubt whatsoever on my identity.

That being said, you are not required to carry an ID at all times, but you are required to present it to authorities who are entitled to control ID when they ask you to, and you must allow them to verify your face matches the photograph (so, no burkha or such).
Authorities, that includes police. Contrary to popular belief, police does not really need a reason to ask you for your ID. Well, they do, but them thinking you’re suspect in some way is enough of a reason. So, while they cannot cannot just indiscriminately ask everybody passing by in the street to present an ID (other than in exceptional situations), they can very well ask you, because you look suspect to them.

Which basically means you should carry an ID anyway, or you may be detained or accompanied to your residence to get your ID.

Update:
Upon re-reading the law again because of a comment below, I noticed a pecularity which suggests that being detained is not the worst that can happen. It seems that in the most perverse sense of the wording of the law, even if you generally own a valid ID, it is still possible that you can be fined those 3,000 Euros.
First, the law literally says “besitzen” (possess) although “eignen” or “zu eigen haben” (own) would, albeit old fashioned and not common language, probably be the accurate wording.
The difference is that “besitzen” implies exercising control, but not necessarily owning something. A thief who steals your car “possesses” the car, while you “own” it. I’m not sure whether this translates well to English, and if the meaning is analogous.

That may, of course, be a mere inaccuracy, and not intentional. In the most perverse sense, it would mean that if I steal your ID (so I possess it), then you are breaking the law, and I’m perfectly good. So that looks like an unintentional mistake. This cannot be the law’s intent.
But note that the paragraph about fines states: “… in violation of paragraph 1 […] does not, or does not in time, present an ID”.
Wow, wait a moment! In time? What does “in time” mean? Who decides what is “in time” and what isn’t? Well, guess who, the official you deal with does, as he likes. See where this is going? You should really have the ID with you, or not too far away, anyway. It’s best not to tempt your luck.
/Update

The remark in a comment above about not inviting Police into your home is correct, and not correct. Police will not normally search your stuff and isn’t entitled to do so. They’re not even entitled to look into the trunk of your car, except if you open it and thus they can obviously look inside (though they may, and will, ask to inspect the first-aid kit, which most people keep in the trunk).
However, they are required by the law to pursue anything illegal they see, and to follow every reasonable, justifiable suspicion of a crime, so it’s nevertheless a very bad idea to let them into your home. You never know what illegal things you might own. No seriously, you don’t know. I cannot even tell whether I broke the law recently because laws are so perverse nowadays.

You are not required to hand your ID or deposit it with someone or anyone (except authorities entitled to control or withdraw ID, which again includes police), and in fact it is illegal for someone to ask you to give up custody of your ID in any way, except for said authorities. So, strictly speaking, what the Post does regularly when you fetch a parcel, or what many hotels do is illegal (since, once your ID is on the other side of the bulletproof counter window, it’s very much no longer in your custody).
Regarding a below comment, it is not truly illegal for e.g. a hotel to copy an ID. Paragraph 20 explicitly permits the owner or someone else with the owner’s explicit consent to make a copy provided that the copy is clearly identifiable as copy. They may, however, not give the copy to a third party and they may not use any of the data on the ID card, except if the owner explicitly agreed (that may be one reason why hotels make you fill out forms with “data that they already know” such as the name, date of birth, and address as written on the ID card).

You may also be asked for an ID when travelling on the train since a month or two (my wife had this happen recently). Used to be the credit card you bought the (personalized) ticket with or your Bahncard (if you have one) was enough as identification, used to be you needed none at all, but now they seem to insist on an ID. Though I’m not sure what would be the consequence if you refused to show one. After all, you have a ticket either way, and technically the conductor is not entitled to ask for ID. But if you prefer a calm stay, you should probably not try it.

As a Tourist

I’d probably take the passport with me.

As a Resident Alien

I’d handle it much as German nationals do with their Personalausweis. If you’re in your hometown and not engaged in any activity that might be misconstrued, you can leave your ID at home. Police actually do need a reason to ask "Papers, please!" but this reason could be that you’re witness to a traffic accident or that you look like someone they are searching for.

• If you are running through a park early in the morning, odds are very low that you will be asked for papers, and it will not be suspicious that you don’t have them. Especially if you can answer in fluent German where you live and this information checks out against the databases of the Bürgeramt.
• If you are slowly strolling through a park known for drug dealing in the evening, better bring your papers with you.
• If you are operating a motor vehicle, take both the license and the ID papers.
• If you are traveling to another city, take the papers along.

Non-Official Purposes

If you go to the post office to collect a parcel, or to a rental company to rent a car, you will know that you’ll need ID. Bring it …

When staying in Germany, you are by law required to be in the possession of a valid and recognized identification document, but not to carry it with you. This applies both to German citizens older than 16 years of age (PAuswG § 1) and to foreigners (AufenthG § 3).

Even if you are not required by law to carry an ID, it can, especially for foreigners, save you a lot of hassle. Unlike in many other countries, German police are, in many situations without any specific reason or suspicion, entitled to stop you and request and verify your personal data. In practice, this means that the police are entitled to detain you until they can verify that you are whoever you claim to be, if you are not carrying any ID documents. Foreigners can also be detained until the police can verify that you are legally present in Germany.

Having lived as a foreigner in Germany for many years and being in a similar situation as you are (not having a recognized national ID card, I ought to carry my passport), the practical consequences depend on whoever you are confronted with. In most situations, the police have been satisfied seeing a driver’s licence, although they by no means are obligated to recognize that as an ID document. I have also a few times been detained for shorter periods.