Under the Visa Waiver program, can I leave by land in transit to Guatemala?

When entering the US by air, can one cross the Mexican border with the intent of leaving Northern America?

Yes, of course.

How does the transit time in Mexico count toward the 90 day limit of the VWP?

Well, if you reenter the US without first being in Central America, the time in Mexico counts against your 90 days. If you do go to Central America, then the time in Mexico does not count against the 90 days.

How does the US register this exit?

It probably doesn't. That's no problem, really; your entry stamp for Mexico serves as proof of having departed the US. If you don't get a passport stamp when entering Mexico, keep your bus ticket or fuel receipts.

The most significant thing is when you next apply to enter the US. If you never try to reenter the US, it doesn't matter at all. Furthermore, your only real problem is if you want to reenter the US sooner than 90 days after your first entry and leave more than 90 days after your first entry. That's because, if you go only to Mexico, the immigration officer has the option of admitting you only until your original departure date. In other words, if you arrive on July 1st and are admitted until September 29th, then go to Mexico and return on September 15th, the officer may decide to admit you until September 29th.

This is governed by regulation; 8 USC 217.3(b) states:

(b) Readmission after departure to contiguous territory or adjacent island.

An alien admitted to the United States under this part may be readmitted to the United States after a departure to foreign contiguous territory or adjacent island for the balance of his or her original Visa Waiver Pilot Program admission period if he or she is otherwise admissible and meets all the conditions of this part with the exception of arrival on a signatory carrier.

If you plan to leave before that date anyway, of course you have no problem. Of course, if you are admitted more than 90 days after your previous entry, then you will get a new period of entry.

To avoid triggering the provision of the law that allows the officer to readmit you for the balance of your previous admission, you have three options:

1. Show that you've been to Guatemala. A Guatemala passport stamp with an appropriate date ought to do that.

2. Re-enter the US more than 90 days after your previous entry.

3. Obtain a US visa.

Finally, there's a question you didn't ask:

Will I be able to use the VWP to enter the US?

As noted in a comment by Henning Makholm, one requirement to use the Visa Waiver Program is to have a round trip ticket. This is a statutory requirement, found at 8 USC §1187(a)(8):

Round-trip ticket

The alien is in possession of a round-trip transportation ticket (unless this requirement is waived by the Secretary of Homeland Security under regulations or the alien is arriving at the port of entry on an aircraft operated under part 135 of title 14, Code of Federal Regulations, or a noncommercial aircraft that is owned or operated by a domestic corporation conducting operations under part 91 of title 14, Code of Federal Regulations).

The relevant regulations are found at 8 CFR 217. 217.2(a) defines "round trip ticket" as

any return trip transportation ticket in the name of an arriving Visa Waiver Pilot Program applicant on a participating carrier valid for at least 1 year, electronic ticket record, airline employee passes indicating return passage, individual vouchers for return passage, group vouchers for return passage for charter flights, and military travel orders which include military dependents for return to duty stations outside the United States on U.S. military flights. A period of validity of 1 year need not be reflected on the ticket itself, provided that the carrier agrees that it will honor the return portion of the ticket at any time, as provided in Form I-775, Visa Waiver Pilot Program Agreement.

The regulatory implementation of the round-trip ticket requirement is found at 217.2(c):

(c) Restrictions on manner of arrival -

(1) Applicants arriving by air and sea. Applicants must arrive on a carrier that is signatory to a Visa Waiver Pilot Program Agreement and at the time of arrival must have a round trip ticket that will transport the traveler out of the United States to any other foreign port or place as long as the trip does not terminate in contiguous territory or an adjacent island; except that the round trip ticket may transport the traveler to contiguous territory or an adjacent island, if the traveler is a resident of the country of destination.

(2) Applicants arriving at land border ports-of-entry. Any Visa Waiver Pilot Program applicant arriving at a land border port-of-entry must provide evidence to the immigration officer of financial solvency and a domicile abroad to which the applicant intends to return. An applicant arriving at a land-border port-of-entry will be charged a fee as prescribed in § 103.7(b)(1) of this chapter for issuance of Form I-94W, Nonimmigrant Visa Waiver Arrival/Departure Form. A round-trip transportation ticket is not required of applicants at land border ports-of-entry.

In practice, it seems that the enforcement of this requirement is sporadic. The most likely point where it would cause trouble is that the airline may not board you if you have only a one-way ticket. You can get around that requirement by obtaining a US visa.