‘Avoid sleeping on your back’ & ‘breathe in toilet smells’ were seen as precautions against the Black Death. Why did doctors think these would work?

This cannot be answered comprehensively here. But a few key points might be revealed. The precautions mentioned in the question are by far not the only ones that were given, just mere examples.

At the time, medical and lay authorities throughout Europe sought to give rational explanations for the virulent plague, which was clearly contagious. They issued many treatises trying to explain to the public the causes, symptoms, and treatment of the disease, which was in general blamed on celestial, terrestrial, and miasmatic facts as well as contagion (transference from infected persons). The plague was attributed to any and all of the following: corrupted air and water, hot and humid southerly winds, proximity of swamps, lack of purifying sunshine, excrement and other filth, putrid decomposition of dead bodies, excessive indulgence in foods (particularly fruits), God’s wrath, punishment for sins, and the conjunction of stars and planets. [emphasis added]

— George Childs Kohn: "Encyclopedia of Plague and Pestilence: From Ancient Times to the Present", Facts On File: New York,^3rd2008, p31–33.

The prevailing theories to consider are hippocratic humoralpathology (humorism); that is: imbalances of juices cause illness and re-balancing them cures the illness. Applied to the theory of miasmas that means avoiding bad air is best, but using good smells can also re-balance the causative principle of the badness. Toilet smell is just a subtype of miasma (roughly "bad airs") coupled with the widely held belief that food is concocted and rotting in the process of digestion. The Black Death is indeed partly airborne when the infection is bubonic plague with lung involvement.

Concerning to actually "breathe in toilet smells to avoid the plague": this was indeed a desperate attempt to fight fire with fire. But a rare one. The principle of "And whoever smells you also gets it." (p51) was the prevailing theory adhered to. — Samuel K. Cohn: "Cultures of Plague: Medical Thinking at the End of the Renaissance", Oxford University Press, Oxford, 2010; listing countless examples across Europe for bad smells to avoid.

This theory in favour of bad smells had some supporters of a certain calibre:

The smells did not have to be sweet; any strong smell might serve the purpose. Smoking or chewing tobacco was thought to bring immunity. In the 18th century it was said that the cess-pits in London had been opened during the Great Plague in an attempt to ward off the disease. In 1665 Dr Francis Glisson recommended as his ‘constant antidote’ a piece of dried manure of someone who had died of the disease, kept in a house in a porous box for ‘the best antidotical perfume’. Many of the nostrums and antidotes were unconvincing, and this one in particular sounds like the advice of a crank or at least an eccentric, but Glisson was a Cambridge man, who was Regius Professor of Physic for more than 40 years, a Fellow of the College of Physicians and an original fellow of the Royal Society.

— Stephen Porter: "Disease and the City 17th Century: Plague", Gresham College Lecture, 2001.

This was a minority opinion and only fashionable for a time.

By far the most widely accepted cause of the plague, however, was foul odour caused by putrefaction. In the words of a medieval alchemical poem: "And when evyl substance shal putrefie Horrible odour // are gendered therbye // As of dragons, and men that long dead be, // Their stinke may cause great mortalitie."

— Constance Classen, David Howes and Anthony Synnott: "Aroma — The cultural history of smell", Routledge: London, New York, 1994, p59.

But also note the emergence of principles as forerunners of Paracelsus and Hahnemann's homeopathy (similia similibus curantor):

In fact, the fumes may not have purified “corrupted air,” but they would have acted as a stimulant and made a patient feel better. Others believed in battling stink with stink, a variation on the poison-versus-poison tactic advocated by later Paracelsians. This distinctly minority view advocated the use of nasty-smelling matter, including buckets of human waste, or the burning of foul-smelling materials like sulfur, leather, gunpowder, or human hair. Fire alone was also considered effective in purifying the air, whether in or out of doors. In 1348 papal physician Guy de Chauliac had Clement VI sit between two great fires in the main hall of his palace in Avignon for this very purpose. Fire’s natural drying effect counteracted the vile humidity that, he believed, sustained the corruption.

— Joseph Patrick Byrne: "Daily life during the Black Death", Greenwood Press: Westport, 2006, p49.

There were mono- or combined applications of different medicinal plants. Approaches that we know from homeopathy were also celebrated at that time, consciously or unconsciously: "Stinking roots expel the stinking plague" (expel like with like). This also applied to the root of valerian (Valeriana officinalis L.).

— Ellen Wulfers: "Heilpflanzen als Mittel gegen die Pest im Mittelalter und in der frühen Neuzeit*, Schweiz Z Ganzheitsmed 2014;26:34–44. DOI: 10.1159/000358109 (my t)

More in line with current thinking would of course be the notion that once you combine bad aroma as cause for illness with the insight that for certain illnesses some survivors develop a level of immunity. It can hardly be argued that the concept would have been totally new to medieval doctors:

The Athenian epidemic of 430-426 BC, […] In his remarkable documentation of the epidemic, Thucydides (who survived the disease) not only left us a clear clinical picture of the pestilence but also identified its infectious nature and the fact that it conferred at least partial immunity on survivors.

This should not be read as as some form of thinking that lead to vaccinations but more like some kind of resistance training: What doesn't kill you makes you stronger.

Counteracting bad with bad is largely discredited by now, but it is not entirely unreasonable for seeing the deficiencies of effect of the other guidelines and then trying to do just the opposite. While those remedies that are outlined in this answer and which might be read as "they work": they did not always work and 'mis-applying' some apparent principles will have added to that.

Sleeping on your back was thought to be associated with sudden infant death – and is now associated with late still births. A nice empircal observation is that people primarily sleeping on their back tend to snore, get apneas and do not wake up as rested as others. Sleeping on one's back is therefore generally unhealthy and best to be avoided as a general health advice.

Bad air, the wrong type of food, lack of sleep, too much blood, or fright could affect the balance of humors and cause disease. Relocation to a place with good air, a diet of the right food, sound sleep, phlebotomy (controlled bleeding), or soothing music could heal. This matrix was the heart of Galenic medicine, which served as the model for Arabic, Byzantine, and western European medical theory and practice over nearly two millennia.

— Byrne/Aroma, p 17, emphasis added.

Notice that 'sweet smells' is what inspired the design of this mask:
This is not intended as a particulate filter but the beak is filled with herbs, spices and oils. An unintended consequence of this gear is that not only the transmission vector of fleas carrying the Yersinia pestis have a harder time to get to the wearer. Another is that also the directly airborne infectious droplets are somewhat filtered and some of the ingredients providing aroma might actually have had a bit of antibacterial like effect. The plague doctor Notradamus is reported to have used a mixture consisting of rose petals, cypress, iris, cloves, calamus, tiger lily and aloes. Tiger lily as an antibiotic? Another unintended discovery might be possible there.

tl;dr

Any religious significance might be added on top of that. But the core is a long standing tradition of empirical science. There was the observation, there was the theory and it often seemed to work according to theory. _{(Please do not judge this on any post-Galilaen concepts, it's history of science.)} There was some apparent contemporary evidence that this worked and there is modern evidence that the advice given in these writings was not entirely hopeless, despite the desire to paint it as peculiar and even ridiculous.