This looks pretty much like a conflation of the type already mentioned in comments:

Are you, perhaps, thinking of Samuel Johnson's dictionary, where he described oats as: "a grain, which in England is generally given to horses, but in Scotland supports the people"?
(—@sempaiscuba)

A few indicators to support this assumption:

• Such a description is surely hyperbole. So while perhaps a nice quote of derogatory nature, it would not be of much use to inform anyone about the 'situation on the ground'.

• If it would be about Germanic people, not many authors come to mind, mainly Caesar and Tacitus. Both authors didn't reveal to my searches such quips. Tacitus reveals barley as typically Germanic for beer-making. (Which would explain an actual difference: Romans preferring wine over Germanic beers)

  XXIII Their drink is a liquor made from barley or wheat, fermented so as somewhat to resemble wine. The frontier tribes do indeed buy wine. Their food is simple : wild fruits, fresh game, or curdled milk ; they appease their hunger without luxurious accessories to tickle their senses. In quenching their thirst they are by no means so temperate ; let them but be given all the intoxicating liquor they choose to drink, and vice will make an easier conquest of them than the sword.

  In Tacitus Germania there is the general description of

  V Although the land is somewhat varied in appearance, nonetheless on the whole it is gloomy with forests or unwholesome with swamps, damper where it faces the Gauls, windier where it faces Noricum and Pannonia; it is fertile for grain crops, does not bear fruit trees, and is rich in livestock, but the animals are generally small.

  for which we get the following explanation:

  satis, ablat. pl. of substantive; "fruitful in crops,' e.g. barley, wheat (13.1), and oats, the principal German cereal (Pliny, N.H. XVIII. 44.149). The cultivation of the first two in Germany goes back to Neolithic times, of the last to the Bronze Age (see Schwyzer ad loc.).
  — Tacitus Cornelii Taciti de Vita Iulii Agricolae, De origine et Moribus Germanorum)

• If about Germanic people, rye in the first century would be extremely rare, growing more like weed in fields, but they did have barley and wheat, although both not much for baking, having more puls gruel-like meals than bread.

• If about Celts, in a German Hessian saline a find from 713 BC (dendro dating of AKl4 BNau Kurstraße Quadrant N9, Planum Hc, 2002) was identified of being baked in a region without any rye or oats, and composed of 'some' wheat variety (barley present in the area but incompatible with excellent fully leavened fluffiness indicating maximum gluten content)
  — Andreas G. Heiss & Angela Kreuz: "Brot für die Salinenarbeiter - das Keltenbrot von Bad Nauheim aus archäobotanischer Sicht", Hessen Archäologie 2006 – Jahrbuch für Archäologie und Paläontologie in Hessen, 2007.

• Also, for any 'lower' group of people the saying goes 'civilised people eat wheat, but' insert slaves, soldiers to be disciplined, and gladiators ate mainly 'fattening' barley. So much so that any such group had the term hordearii (barley eater(derogatory)) attached as 'nickname', especially the gladiators. Recall similar sayings for Athens and Sparta.

• and:

  THE CHOICE OF GRAINS

  There is a general tendency in historical literature to overestimate the role played by wheat in classical antiquity relative to the other grains. A reader is easily misled into believing that wheat was the most dominant grain all during the classical era. For example, J. J. VAN NOSTRAND (7) took the evidence pertaining to grain in general, and to barley specifically, and applied it to wheat with the result that wheat was presented as being practically the only grain grown and consumed in Spain. A. C. JOHNSON (8) assumed that wheat was the only grain used for food in Egypt. The per capita wheat allowance for Sicily is so large in V. M. SCRAMUZZA's (9) study that practically no room is left for the consumption of barley. AUGUSTE JARDE' (io) computed for Attica the requirements for barley as feed at a figure as high as the total output of this grain, and thus was forced to the conclusion that, except for imports, only the output of wheat was available for human consumption (i i); this would clearly imply that the owners of estates in which barley was the only grain grown purchased imported wheat for their slaves.

  If there is one topic on which no progress has been made in the last two millennia, it is the persistence of PLINY'S idea that the Greeks liked barley while the Romans preferred wheat, which is repeated to the present day. But, even now, only a minority of the earth's population can afford to use as food that kind of grain which it prefers: wheat in the Occidental, and rice in the Oriental world. The rest make use of that grain which can be produced locally with the smallest amount of labor or on the smallest area of land. In addition to wheat and rice, huge quantities of rye, corn, grain-sorghum, millet, buckwheat, barley, and oats are consumed for food in the world-for all practical purposes because the people can not afford the grain preferred by them.

  — Naum Jasny: "The Daily Bread of the Ancient Greeks and Romans", Osiris, Vol. 9, 1950, (p227–253).

So while the Romans much preferred first emmer, and continued to have a special place in their heart for it that never disappeared, they started to dislike pure barley. In reality all grains available were grown to suitable soils and all were eaten, in various combinations. Emmer, spelt, wheat, barley were all mixed into various forms of panis.

For one Germanic region that became a Roman frontier hinterland the actual archaeological evidence looks like this:

The Middle and Late Iron Age (500 BC—19 BC)

The crops grown by the Iron Age farmers were emmer wheat (Triticum dicoccum), spelt wheat (Triticum spelta), hulled multi-rowed barley (Hordeum vulgare var. vulgare), oats (Avena sativa), broomcorn millet (Panicum miliaceum), pea (Pisum sativum), horsebean (Vicia faba var. minor), flax (Linum usitatissimum), gold of pleasure (Camelina sativa), rape seed (Brassica rapa), and opium poppy (Papaver somniferum var. setigerum) (Bakels and van der Ham 1980; Bakels et al. 1997).[…]

The arrival of Roman troops altered this situation. The army people, and especially their officers, could obviously not live without their customary foods. Immediately after the Roman invasion exotic plants turn up in the archaeological records, a phenomenon observed in many European sites which before were situated well outside the sphere of Mediterranean influence (Bakels and Jacomet 2003). The civilian administrators who followed shortly after, shared the culinary demands of the army people. [Talking mainly about wine, spices, nuts and vegetables though, LLC …]

One site revealed two grains of bread wheat (Triticum aestivum), prompting the thought that another cereal was added to the list of staple crops, but this is probably a false conclusion. This bread wheat was found in Hoogeloon, which is the site with the highest level of Romanization and presumably the richest inhabitants of the region (van Beurden 2002b). Bread wheat is a rather exacting cereal and does not do well on the local sandy soils. It is a wheat considered always and everywhere as a kind of superior cereal and was probably a luxury in Brabant. It may have been imported from other parts of the Roman Empire, upstream the river Meuse for instance, where conditions for wheat growing were optimal.

— Corrie Bakels: "Crops grown on the sandy soils of Eastern Brabant (the Netherlands) before, during and after the Roman occupation", Analecta Praehistorica Leidensia, 41, 2009, (p57–72). (PDF)

One author that does express a similar sentiment is Pliny:

XIV. Barley is the oldest among human foods, as is proved by the Athenian ceremonya recorded by Menander, and by the name given to gladiators, who used to be called ‘barley-men’. Also the Greeks prefer it to any other grain for porridge. There are several ways of making barley porridge: the Greeks soak some barley in water and then leave it for a night to dry, and next day dry it by the fire and then grind it in a mill. Some after roasting it more thoroughly sprinkle it again with a small amount of water and dry it before milling; others however shake the young barley out of the ears while green, clean it and while it is wet pound it in a mortar, and wash it of husk in baskets and then dry it in the sun and again pound it, clean it and grind it. But whatever kind of barley is used, when it has been got ready, in the mill they mix in three pounds of flax seed, half a pound of coriander seed, and an eighth of a pint of salt, previously roasting them all. Those who want to keep it for some time in store put it away in new earthenware jars with fine flour and its own bran. Italians bake it without steeping it in water and grind it into fine meal, with the addition of the same ingredients and millet as well.

XV. Barley bread was much used in earlier days, but has been condemned by experience, and barley is now mostly fed to animals, although the consumption of barley-water is proved so conclusively to be veryBarley water. conducive to strength and health: Hippocrates…

— Pliny the Elder, Natural History

As well as his take on oats being a new type of grain previously unknown to Romans:

The history of oat domestication parallels that of barley (Hordeum vulgare L.) and wheat (Triticum spp.), the primary domesticated cereals of the Middle East. The primacy of wheat and barley in the Neolithic revolution was due to advantages that their progenitor species had over other local candidates, such as A. sterilis L. and A. longiglumis Dur.: local abundance, large seed weight and volume, absence of germination inhibitors, and lower ploidy levels (Bar-Yosef and Kislev 1989).

In the archaeological record, wild oat appears as weedy admixtures in cultivated cereals prior to, and for several millennia following, the Neolithic revolution. Nondomesticated Avena spp. have been identified in archaeological deposits in Greece, Israel, Jordan, Syria,Turkey, and Iran, all dating from between about 10500 and 5000 B.C. (Hopf 1969; Renfrew 1969; Hansen and Renfrew 1978; Hillman, Colledge, and Harris 1989).

Wheat and barley remained predominant as cereal cultivation spread through Europe between the seventh and second millennium B.C. (Zohary and Hopf 1988). The precise time and location of the domestication of oat from the weedy component of these cereals is unknown, but it is believed that oat had an adaptive advantage (over the wheat and barley germ plasm in cultivation at that time) in the cloudier, wetter, and cooler environments of northern Europe.

Support for this theory is provided by Pliny (A.D. 23–79), who noted the aggressive nature of weed oat in cereal mixtures in moist environments (Rackham 1950). Z. V. Yanushevich (1989) reported finding Avena spp. in Moldavian and Ukrainian adobe imprints dated as early as 4700 B.C. It is not known if these were cultivated types. However, Z. Tempir, M. Villaret-von Rochow (1971), and U. Willerding (Tempir and the latter are cited in Zohary and Hopf 1988), working in central Europe, found evidence of domesticated oat dating from the second and first millennia B.C. That evidence (which is often a reflection of one of the first steps in the domestication of oat and other cereals) is the elimination of the seed dispersal mechanism. In domesticated oat, the spikelets remain intact on the plant long after ripeness, whereas in the wild species, spikelets abscise and fall from the plant soon after maturity (Ladizinsky 1988).

In China, oat has been cultivated since early in the first millennium A.D. It remains a staple food in north China and Mongolia (Baum 1977), and the hull-less or “naked” oat has been associated with the Chinese production. But despite the cultivation of oat elsewhere, the grain was of minor interest to the Greeks, who considered it a weed; moreover, Egyptian foods are not known to have contained oat and, unlike so many other foods, there is no reference to it in the Bible (Candolle 1886; Darby, Ghalioungui, and Grivetti 1977; Zohary 1982).

During the first century A.D., however, Roman writers began making references to oat (White 1970). Pliny described a fall-sown, nonshattering “Greek-oat” used in forage production and noted that oatmeal porridge was a human staple in Germany. Dioscorides described the medicinal qualities of oat and reported it to be a natural food for horses (Font Quer 1962). Analyses of the gut contents of a mummified body from the same era (recovered from an English bog) revealed that small quantities of Avena species, together with wheat and barley, were consumed in the final meal (Holden 1986).

Although archaeological records indicate that primitive peoples employed oat as a food source, the first written reference to its use was Pliny’s observation that the Germans knew oat well and “made their porridge of nothing else” (Rackham, H. 1950. Pliny natural history, Vol. 5. Books 17–19. Cambridge, Mass.). Oatmeal porridge was an acknowledged Scottish staple as early as the fifth century A.D. (Kelly 1975). Porridge was prepared by boiling oatmeal in water, and it was consumed with milk, and sometimes honey, syrup, or treacle (Lockhart 1983).

— Kenneth Kiple: "The Cambridge World History of Food", Cambridge University Press: Cambridge, New York, 2000.

Conclusion

Roman knowledge of grains did change when they learned something new at roughly the time of conquering parts of Germania. But it is unlikely that these were either rye or barley. While barley was of somewhat lower culinary esteem, it was known to be nutritious and healthy, and used by almost all classes.

Similar for Germanic tribes which used wheat and barley, but still not rye for everything, but barley being most prominently mentioned by Roman authors for its role in beer brewing.

What was new and 'Germanic' for Roman authors were spelt and oats (Lloyd Thomas Evans, L. T. Evans, W. J. Peac**k: "Wheat Science - Today and Tomorrow", Cambridge University Press, Cambridge, New York, 1981, p 11).

That leads to the inference that it might be a quote more about peculiar Germans eating pure oat gruel, and not really directly connected to any type of Roman bread. Since Pliny write Nat.Hist 18,44 that 'oats are a weed that spoils the other grains' and still the German eat it out of preference, this seems the most likely?

The foremost feature of disease in wheat is the oat. Barley, too, will degenerate into the oat; so much so, in fact, that tile oat has become an equivalent for corn; for the people of Germany are in the habit of sowing it, and make their porridge of nothing else. This degeneracy is owing more particularly to humidity of soil and climate; and a second cause is a weakness in the seed, the result of its being retained too long in the ground before it makes its appearance above it. The same, too, will be the consequence, if the seed is decayed when put in the ground. This may be known, however, the moment it makes its appearance, from which it is quite evident that the defect lies in the root. There is another form of disease, too, which closely resembles the oat, and which supervenes when the grain, already developed to its full size, but not ripe, is struck by a noxious blast, before it has acquired its proper body and strength; in this case, the seed pines away in the ear, by a kind of abortion, as it were, and totally disappears.

The wind is injurious to wheat and barley, at three periods of the year in particular: when they are in blossom, directly the blossom has passed off, and just as the seed is beginning to ripen. In this last case, the grain wastes away, while in the two former ones it is prevented from being developed. Gleams of sunshine, every now and then, from the midst of clouds, are injurious to corn. Maggots, too, breed in the roots, when the rains that follow the seed-time are succeeded by a sudden heat, which encloses the humidity in the ground. Maggots make their appearance, also, in the grain, when the ear ferments through heat succeeding a fall of rain. There is a small beetle, too, known by the name of "cantharis," which eats away the blade. All these insects die, however, as soon as their nutriment fails them. Oil, pitch, and grease are prejudicial to grain, and care should be taken not to let them come in contact with the seed that is sown. Rain is only beneficial to grain while in the blade; it is injurious to wheat and barley while they are in blossom, but is not detrimental to the leguminous plants, with the exception of the chick-pea. When grain is beginning to ripen, rain is injurious, and to barley in particular. There is a white grass that grows in the fields, very similar to panic in appearance, but fatal to cattle.

The above is largely compatible with what's depicted in –– Heinrich Eduard Jacob (Translated By Richard & Clara Winston): "Six Thousand Years of Bread: Its Holy and Unholy History", Doubleday: New York, 1944.

The crops grown by the farmers of northern Germania are well established in a variety of sources, chiefly pollen spectra and finds of seeds and other plant remains in dated deposits. Barley had been grown in quantity from the second millennium bc and it was to continue as the most prevalent form of grain crop until the early medieval period, especially the hulled varieties. Oats were also widely grown, having been originally collected as a weed among cultivated grain. Various forms of wheat, including emmer and Einkorn, were widely cultivated, as they had been since the Neolithic. Rye and millet appear in varying quantities, as do other seed-bearing plants such as gold-of-pleasure.

The abundance of barley among the cultivated grains of the north may be partly explained by its use in the making of beer.

— Malcolm Todd: "The Early Germans", The Peoples of Europe, Blackwell: Malden, ²2004.

Sidenote: The most interesting of these is gold-of-pleasure (Camelina sativa), widely used in Germanic lands before the Romans came, then almost disappearing in newly colnised Germanic lands, extremely good source for nutrients, omega3 fatty acids, now just used as fighter-jet-fuel.