How do young earth creationists explain why domesticated animals are inefficient and have better alternatives?

The primary answer is the Fall of Man. When Adam and Eve sinned, death entered the creation, as well as the slow accumulation of genetic defects, plagues, changes to climate, etc. The plants and animals we have today are cursed. They preserve some of the original perfect design, but with deficiencies.

According to YEC, prior to the Flood, all creatures were vegetarian, so meat consumption was not a design goal. Only after the flood, in a world where good nutrition became immensely more difficult, did God concede to humans and animals the necessity of carnivorous behavior.

Compare the energy density of animal fat (35) to gasoline (34.2). See https://en.wikipedia.org/wiki/Energy_density

Our World economic system currently is based on hydrocarbon consumption, so it is a pretty efficient fuel source, stable and not prone to explode (except in movies). Animal fat is marginally more efficient. I would call that excellent design.

Addition:

In God's speech to Job, he reveals part of the reason for his creation of the animals. They were not created solely for man. God derives delight from all his creations. Also, if you look at the different type of care that God shows toward each listed animal, you find every major physical, emotional and spiritual need represented, either directly or by parabolic analogy. The weakness or neediness of the animals shows forth God's character, as he lovingly meets their needs. Among the needs listed is help in delivering and raising young.

“Do you know when the mountain goats give birth?
    Do you watch when the doe bears her fawn?
2 Do you count the months till they bear?
    Do you know the time they give birth?
3 They crouch down and bring forth their young;
    their labor pains are ended.
4 Their young thrive and grow strong in the wilds;
    they leave and do not return." (Job 39:1-4)

since rapid speciation is a thing for young earth creationists, there is no reason why the domesticated animals humans have should not evolve at hyper speed to more sane for purpose forms

This statement is false, and demonstrates a misunderstanding of YEC.

Animals don't evolve. There is exactly zero evidence of evolution (that is, appearance of useful de novo features, much less the ability for an organism to randomly mutate into an entirely different organism), period. (Whether or not you believe a priori in YEC.) "Evolutionism" is in direct contradiction to the laws of thermodynamics / entropy. Evolutionists claim to observe "evolution" because their worldview falls apart otherwise. For a fun project, though, do some digging and see how much "evidence" you can actually turn up that isn't explainable by natural selection, and how many "unsolved" problems exist.

Animals exhibit natural selection according to a pre-existing genetic potential with which they were created, which enables them to adapt to changing environments. As time goes on, however, this information is gradually being lost. The key, however, is that animals cannot "evolve" in any way that wasn't already encoded at Creation.

As Paul notes, carnivory was not part of the original Creation. While it seems that God likely provided for it, it is still dubious to assume that God created any animals to be ideal for humans to raise for meat. Rather, via domestication, we try to squeeze existing animals into that mold (via artificial selection), but we are limited to the genetic potential of animals as they were created.

That all being said, it's worth keeping in mind that modern meat production is just that; modern. Historically, people raised chickens mostly for eggs (and they are actually pretty good at egg production!), and kept cattle and goats for milk and as draft animals. (Sheep I think are still mainly raised for wool.) Slaughtering them for meat was done either out of necessity (because you are starving and don't have a choice), or as a way of making use of an animal that was too old or too injured to continue its original "job". There are a number of factors other than pure efficiency that guide modern meat production. If efficiency was the main concern, we'd probably be eating insects of some sort.

That said, I think your question is also lacking citations. For example, while reptiles may require less calories to grow, I believe they tend to grow much more slowly and are rather "delicate", prone to many health problems unless extreme care is taken in their handling. In fact, if there were animals (other than insects, which have cultural issues) that were "better" for meat production, I would expect that we would be raising those other animals.

Also, extinct animals that young earth creationists believe were with humans have members that have no reason to not just be better than modern animals for domestication. For example, non-avian herbivorous dinosaurs [...]

For starters, those animals are extinct. Importantly, the reason why they are extinct is likely relevant. Many YECs theorize that the planet's climate/environment changed significantly due to the Flood, which may have had significant deleterious effects on those species we now call "dinosaurs". If those species have significant difficulties just surviving, that would make them less-than-ideal for domestication.

That said... there is possible (albeit highly contested) evidence that dinosaurs were domesticated in the past.

Why would rapid speciation be a thing for wild animals and not domestic ones?

This actually deserves, and has, a serious answer... which is that rapid diversification is absolutely a thing among domestic animals. Domestic animals, or even those being selectively bred by humans, do exhibit rapid changes. (Just look at the last 100 years of dog breeding, or the significant changes that have occurred over a similar period in cattle and domestic fowl, or the Russian project to domesticate foxes.) Human artificial selection can actually act faster than natural selection.

The reason we don't see rapid speciation is because humans have this annoying tendency of preventing, and even reversing it. In the wild, speciation occurs when groups of animals become sufficiently separated, either geographically or behaviorally, to stop interbreeding. If dogs were wild animals, Great Danes and Chihuahuas would absolutely be separate species. For that matter, we consider dogs, wolves and coyotes to all be separate species, despite that all three can and do interbreed, and are probably more genetically similar than many breeds of domestic dog. However, we humans are particularly adept at encouraging animals that otherwise would be separate species to interbreed, thus preventing "true" speciation. And then we go and breed animals that are supposed to be different species, like lions and tigers, or horses and donkeys. (Both of which have produced fertile hybrids... and servals crossed with domestic cats are very successful. Really, the whole concept of "speciation" is somewhat problematic, at least if we're trying to assert a difference versus breeds within a species... or kind.)

there is no reason why the domesticated animals humans have should not evolve at hyper speed to more sane for purpose forms

The problem with this assertion is that it presumes to know God's purpose. In fact, this is exactly what we see... if we accept that "more sane for purpose" is according to God's plan, and not ours. Yes, animals were originally created for the benefit of humans, but that plan has been altered twice; first, with the Curse, and second following the Flood. We should particularly look at the Curse, since plants were also originally created for animals, and were subsequently changed to bring forth "thorns and thistles" (Genesis 3:18), which are not for our benefit. It is an act of supreme hubris to believe that we know God's intent behind the potential variability in animals today.