What is the penalty for US citizens entering/leaving the US on a foreign passport?

I think the requirement for US citizens to bear a US passport is so that you can prove citizenship on any subsequent re-entry attempt. If you left the US without a US passport, authorities might not believe you are a citizen and might not let you back in, especially not as easily or quickly as you’d like. Pointing to this law, immigration agents (or others you need to convince of your ability to enter, such as an airline/common carrier’s agent) can say it’s your responsibility to have the passport. Anything they do to help someone who has failed to meet this responsibility is at their option and on a timeline convenient for them, not necessarily convenient for the traveler.

Thus, the penalty is the possibility of denied or delayed entry into the US, as well as the associated costs (e.g. missing connecting flights or meetings). You may also need to pay costs associated with obtaining documents proving citizenship urgently for presentation to border agents, if they do not believe your assertion of citizenship.

In addition to entry into the citizen’s home country, having the passport is also often necessary to obtain certain consular services abroad. By not having one’s passport as required by law, such services may be delayed or denied, and there may be practical costs/consequences associated with that.

The law requiring US citizens to "bear" a US passport when leaving and entering the US is the Immigration and Naturalization Act section 215, found at 8 USC § 1185, Travel control of citizens and aliens, subsection (b):

(b) Citizens
Except as otherwise provided by the President and subject to such limitations and exceptions as the President may authorize and prescribe, it shall be unlawful for any citizen of the United States to depart from or enter, or attempt to depart from or enter, the United States unless he bears a valid United States passport.

Source: https://www.law.cornell.edu/uscode/text/8/1185

The limitations and exceptions generally comprise things like frequent-traveler program cards and trips to Canada (the latter exception is now only available to children who are traveling by land). I believe there’s an exception for military personnel traveling on orders, too.

Your questions:

1. What is the maximum prescribed penalty for violating these rules?

There is none. The law originally provided for a fairly stiff penalty, but it also originally applied in times of war. It also originally said "a valid passport" without specifying that it had to be a US passport. Over the years, the law was modified to its present form; the penalty provision was repealed in 1978.

There’s an interesting history of this section at http://isaacbrocksociety.ca/2013/05/01/the-history-of-the-requirement-that-u-s-citizens-only-use-u-s-passports-to-enter-the-u-s/

2. Are US immigration officials known for actually applying the penalty?

Well, no, because there isn’t one. Furthermore, of course, US citizens have a legal right to enter the United States, so CBP immigration inspectors will admit them even when they don’t have the proper documents, as long as the inspector is convinced that the applicant is a US citizen. It appears to be official policy that the entering citizen be advised of the necessity of having a passport and then admitted to the country.

I’m unaware of anything that documents current policy, but a 2006 manual for Customs and Border Protection Field Inspectors — the officers who carry out immigration inspections — reads thus:

12.5 United States Passport Waivers.

(a) General. Although primarily charged with the responsibility of determining citizenship, you are required to verify the validity of a United States passport when one is required by law. When an applicant fails to present a passport or presents an expired document, the immigration officer shall, if satisfied that the person is a United States citizen, advise the individual of the necessity of having a valid U.S. passport. Although technically you are waiving the passport requirement for the Department of State, no form need be completed. In addition, there is no fee collected by INS. (Paragraph (a) revised 10/21/98; IN99-02)

(Source: https://www.shusterman.com/pdf/cbpinspectorsfieldmanual.pdf. This manual was acquired through the Freedom of Information Act, and as far as I know there is no more recent version publicly available.)

Finally, all of this concerns US immigration inspectors and therefore assumes the traveler has reached the US border. No commercial carrier is likely to board a traveler who doesn’t have appropriate documents, so nobody should expect to be able to fly to the US using (for example) a US birth certificate or US naturalization certificate. If a US citizen is abroad without a US passport, and doesn’t want to or cannot get a new passport, it will be necessary to travel to the US by land or by private boat or aircraft.