Why is Ruth's child in the line of Boaz and not Mahlon?

tl;dr>

Why was it recorded like that?

because the story is making a theological point, not a legal one

Is this the norm or the exception?

the exact particulars of Boaz are exception, but it is based on a normal practice

Is there any other recorded incident in the Scriptures where this was done and the lineage was accorded to the deceased person?

Yes, Judah and Tamar, the lineage was based on the person who acted as the dad, not the specific rules of Deuteronomy.

My points:

1. Levirate Marriage appears elsewhere in Scripture, with similar result

The concept you are referring to is levirate marriage and appears in other interesting stories in scripture. In the situation of Tamar, we have the same genealogical "problem." Here we have a woman who was destitute and desperate after the death of not one but two husbands, and the withholding of a third. The child that was the offspring of ultimately Judah and Tamar should have also been ascribed to Er - although the fact that Judah, the father, was fulfilling the job that either Er or Shelah, the brothers, should have done may be a complicating factor. Nonetheless, when Jesus' genealogy is recorded, both the women and the men are called out.

2. Levirate Marriage was about property moreso than a "name"

As moderns, our first look is probably at the "honor" being preserved in the idea "that his name be not put out of Israel," but in ancient times, there was a more practical reason as well - money.

Note that levirate marriage had a property purpose that may be obscured by the phrase "that his name be not put out of Israel." Women in Ancient Israel had no property rights, and absent a child, were destined for a life of destitution. Anna, for example outlived her husband by as much as 80 years, and was basically a homeless woman living in the Temple as a result.

That this was a property thing is attested to in the Lexham Bible Dictionary. Under Levirate Marriage, it says:

Levirate marriage was a cultural expectation designed to preserve some aspect of a deceased’s man’s life. Such marriages could preserve familial inheritance and patrilineal descent, as well as provide support for a widow. The impetus for such marriages could range from moral obligation to legislated responsibility.

Many ancient Near Eastern societies practiced levirate marriage, including Babylon, Assyria, the Hittites, Nuzi, and Ugarit. The custom was also present in parts of modern Africa and Asia (Falusi, “African Levirate,” 300–08). In much of the ancient Near East, the goal of such a practice was to preserve a family’s inheritance. While provisions existed allowing women to inherit property, and though polygamy and adoption offered increased chances of progeny, varying circumstances created problems to which levirate marriage offered a solution (Davies, “Rights,” 138–41).

It should also be understood that levirate marriage was as much a "tradition" as it was a law. As such, the concept will necessarily lean on the "spirit of the law," moreso than the letter. The Word Biblical Commentary states:

Campbell observes that we must not be misled by such terms as “law-code” and “legislation” into thinking that collections of ancient case law functioned as codes of law do in modern Western countries. We must not regard the OT codes of law as comprehensive and all inclusive, intended to regulate the legal needs of life by a system of courts, lawyers, judges, and police. On the contrary, legal decisions were made by the town elders on the basis of local legal precedents, preserved primarily orally.

With oral preservation and a focus on the story rather than the "law" the fact that Boaz is simply way more important than Mahlon, the practical answer becomes obvious.

3. There is a theological point being made, not a legal one

The question then becomes, why would the biblical record gloss over the property implications? Here, I posit a theological answer, because the theology behind the genealogy is the more "important" lesson for posterity.

In Jesus' "line" four women are noted. None of them have a great reputation:

1. Tamar - well, we;ve already mentioned the incest and deception
2. Rahab was a prostitute. Period. Says so.
3. Ruth was at very least, quite forward, as https://hermeneutics.stackexchange.com/questions/1391/did-ruth-uncover-boaz-feet-or-something-else points out
4. Mary was an unwed teenage mother. Yes, she was given a child by the Holy Spirit, and I belive in the Virgin Birth, but the only people who actually knew that were Mary and Joseph.

In each of these instances, a story is being told about the moms and dads - and they belong more in the "adult" section than as children stories. But therein lies the point. Jews and Christians don't whitewash their history. Doing the official "so that the name isn't blotted out" is a little bit of a whitewash. By recording the "true" stories of the true fathers, the genealogical story retains its significance and the property support that levirate marriage was supposed to attain is also fulfilled.

Furthermore, by "adopting" different fathers into the line, a point about the doctrine of adoption is made for later Christian readers subscribing to a Pauline interpretation.

4. At least some commentators don't think Boaz was actually doing a levirate marriage, but if he was, his role is correctly reported.

Excursis. The Word Biblical COmmentary has some additional insights:

(1) Ruth 4:5, 10 show that Boaz’s marriage to Ruth was a levirate type of marriage, similar at least to that prescribed in Deut 25:5–10. Even though Boaz was not a levir (Heb. יבם), a “brother-in-law,” this was a levirate marriage because 4:5, 10 specify that the purpose of the marriage to Ruth was “to raise up the name of the deceased on his inheritance so that the name of the deceased shall not be cut off from his brethren” (4:10), which is similar to the purpose given for levirate marriage (cf. Deut 25:6). (2) In 3:9 Ruth bases her request that Boaz accept the levirate marriage responsibility on the fact that he is a גאל, “redeemer.” Therefore, one of the legal responsibilities of the “redeemer” was to perform the levirate marriage, even though the passages in the OT legal corpora dealing with redemption never touch upon the levirate obligation; nor do the passages dealing with the levirate ever call the one responsible for this obligation a “redeemer.” (3) In chap. 4 land that belonged to Elimelech suddenly surfaces in the story, and Boaz calls upon the redeemer more closely related than he to act upon his rights and duties as redeemer and buy back (or preempt the sale of) this land. Further, the standard interpretation argues, Boaz uses the implications of the redeemer’s double responsibilities of “levirate” marriage to Ruth and of redemption of the land of Elimelech to induce the unnamed גאל, who has prior rights, to cede those rights and responsibilities to him. Boaz then redeems the land and marries Ruth. For representative treatments, see the discussion of Campbell, 132–37; Thompson and Thompson, VT 18 (1968) 79–99, and for a full treatment, see Leggett, The Levirate.

Recently a number of studies have sought to solve the many difficulties that arise in applying the postulates outlined above to the exegesis and interpretation of chaps. 3 and 4 of Ruth by denying that the marriage between Ruth and Boaz is connected in any way with levirate marriage. See in particular Beattie, VT 21 (1971) 490–94; VT 24 (1974) 251–67; JSOT 5 (1978) 39–48; JSOT 5 (1978) 65–68; Gordis, “Love, Marriage, and Business,” 241–64; Sasson, JSOT 5 (1978) 49–64;Ruth, 125–29, 143–46.

And this is an interesting problem, mentioned in

PROBLEM: Deuteronomy 25:5–10 delineates what is known as the law of the levirate marriage. If a man dies and leaves his wife childless, the man’s brother was morally obligated to take his brother’s wife and raise up children in the name of his deceased brother. This practice insured that the brother’s name would not die out. However, Boaz was not the brother of Ruth’s dead husband. Wasn’t this marriage arrangement contrary to the law of the levirate marriage? SOLUTION: Although it was a more complicated incident, the arrangement between Boaz, Naomi, and Ruth was certainly not contrary to the law of the levirate marriage. The primary concern of the levirate marriage was to perpetuate the family line of the deceased. Certain factors indicate that by the time of Boaz and Ruth, some additional provisions had become customary. First, if there was no surviving brother in the immediate family, then the obligation of the levirate marriage became the responsibility of the nearest surviving male relative. In this case, there was one individual who was nearer to Naomi’s family than Boaz. However, when he declined the invitation, Boaz became the nearest male relative and the next in line to legally fulfill this moral obligation. Second, along with the responsibility to raise up children in the name of the deceased, there came the responsibility to redeem any property that belonged to the deceased and that had been sold or forfeited (Lev. 25:25). Because the nearest kinsman was not financially able to assume this responsibility (Ruth 4:6), he relinquished his right and responsibility to redeem Naomi and marry Ruth and passed that obligation on to Boaz. There is nothing in the account of the redemption of Naomi and the marriage of Boaz and Ruth that is contrary to the law of levirate marriage.