Clenching teeth and pressing tongue against roof of mouth

‘Suppose, with my teeth clenched and my tongue pressed against my palate, I beat down, hold back, and crush the mind with mind.’

So, with my teeth clenched and my tongue pressed against my palate, I beat down, held back, crushed the mind with mind. While I did so, sweat ran from my armpits.

It was just as if a strong man, holding a weaker man by the head or shoulders, were to restrain, subdue, and attack him;

even so, with my teeth clenched and my tongue pressed against my palate, I beat down, held back, and crushed the mind with mind, and sweat ran down my armpits.

But although I exerted tireless energy, and unremitting mindfulness was established in me, my body was overstrained and lacking calm, because I was exhausted by the painful striving.

But, Aggivessana, such painful feeling that arose in me did not invade my mind and remain.

Mahā Saccaka Sutta

The above description mentions this as practice to gain unremitting mindfulness but did not achieve its goral. Also this is associated the period of self mortification of the Bodhisattva. When it is generally mentioned in the context of Vitakka Saṇṭhāna Sutta it is recommended as a last resort where all else fail and with a word of caution citing that it was what was used during the harsh ascetic practices.

So to answer you questions. Yes. It is recommended with a strong word of caution. It is not recommended for prolonged practice.

It speculated that since in Anapanasati meditation around the mouth is used and in this case upper pallet is used since there is some significance in this region perhaps due to nerve centers or other reasons. But the Suttas are silent on the exact reason.

Yes I have heard this, was taught by a Theravada teacher in Sri Lanka. I am not sure how it works, in fact I cannot explain in neurological terms. However I know serotonin helps to go back to the thing we want to do, to concentrate on. Serotonin can be increased by physical activity, I do not whether pressing tongue does that, but there must be a mechanism behind. According to the sutta, this method is the last resort to be used when a distraction comes to mind when the practitioner attends to a particular or useful theme.

Having the tongue pressed against the roof is standard practice through all traditions. This is basic meditation advice, part of 9-step preparation for meditation, because of the way the chi channels connect, this is good for one's health and growth.

Clenching one's teeth on the other hand I have only read in the suttas as a "last resort" in eliminating and resisting distractions/temptations. It should be taken as a literary device expressing the Buddha's compassion for our suffering and his urging for us to head straight to what we seek, rather than be side-tracked.

How does it work? Does it stimulate certain nerves?

As Ahmed wrote, it is taught in Tai Chi.

When my Tai Chi teacher taught and explained it, it was in terms of Chi flow. His English vocabulary was small and I don't understand Chinese at all, so he could only give a partial verbal explanation. If I recall correctly though, a fairly constant theme of his was that our Chi tends to be too high in our bodies (e.g. up in our heads), and that we should practice bringing it lower (e.g. to centre it around the dan tien).

Some people (e.g. some Westerners) dispute whether there even is such a thing as Chi, calling it a "pre-scientific" description etc. Asking "How does it work, physically?" might not get satisfactory answers.

For me it might work as a learned compliance technique. Because I practised it (literally 'practice' i.e. deliberately repeating the same 'form' on many occasions), pressing my tongue to the roof of my mouth triggers many associated awarenesses (i.e. thoughts, movements, body-memories); it's a mnemonic which starts a whole check-list, for example, "head up, chin back, shoulders down, back straight, pelvis forward, knees bent, and connect the direction (in or out) of the breath to the direction (pull or push) of the hands". So for me it's (at least) a trigger of learned reflexes.

Similarly, perhaps if (or assuming that) you have practised that way (with the tongue) then using your tongue will trigger a remembered state from your past practice.

For what it's worth, it's a convenient gesture: like putting the palms of your hands together, or touching a thumb to a finger tip, it's almost always available, something that you can do in a moment, whether or not you're sitting down.

It's also a constant physical/sensory reminder: if you do it then you feel it, so it's a feedback loop or a circuit. If your mind would otherwise wander maybe it gives you something to focus your attention on.

The gesture also blocks your mouth. Maybe (I'm guessing) that helps to prevent not only talking but even subvocalizing (of internal monologue), and maybe turn awareness towards breathing.

Maybe too (I don't know, maybe I'm reading the wrong thing into it), in the context of the passage quoted in the OP, maybe it's that even if ever you don't control the arising of "evil, unskillful thoughts" you can at least continue by controlling your tongue and so on i.e. controlling your physical form.

According to Thich Nhat Hanh's "The Heart of the Buddha's Teachings" this is a mistake in transmission: page 13-15 on google books. In reality he was asking us not to do this!

The book quotes the Mahasaccaka Sutta, which contains a passage in which the Buddha said that he tried this technique unsuccessfully.

Even during the Buddha's lifetime, there were people such as the monk Arittha, who misunderstood the Buddha's teachings and conveyed them incorrectly (see the Arittha sutta). It is also apparent that some of the monks who memorized the sutras over the centuries did not understand their deepest meaning, or at the very least, they forgot or changed some words. As a result, some of the Buddha's teachings were distorted even before they were written down. Before the Buddha attained full realization of the path, for example, he tried various methods to suppress his mind, and they did not work. In one discourse (the Mahasaccaka Sutta), he recounted:

I thought, Why don't I grit my teeth, press my tongue against my palate, and use my mind to repress my mind? Then, as a wrestler might take hold of the head or the shoulders of someone weaker than he, and, in order to restrain and coerce that person, he has to hold him down constantly without letting go for a moment, so I gritted my teeth, pressed my tongue against my palate, and used my mind to suppress my mind. As I did this, I was bathed in sweat. Although I was not lacking in strength, although I maintained mindfulness and did not fall from mindfulness, my body and my mind were not at peace, and I was exhausted by these efforts, This practice caused other feelings of pain to arise in me besides the pain associated with the austerities, and I was not able to tame my mind.

Obviously the Buddha was telling us not to practice in this way. Yet this passage was later inserted into other discourses to convey exactly the opposite meaning:

Just as a wrestler ... use his mind to defeat his mind.

Often we need to study several discourses and compare them in order to understand which is the true teaching of the Buddha.