How serious was Fermat's statement about the ancients?

Regarding Mathematics, Fermat was sincere and correct.

Certainly it was true that Fermat's contemporaries had a wealth of inventions unavailable to the ancients - gunpowder and the printing press being impossible to ignore. But in mathematics, it is an absolute truth that the ancients knew more than the "moderns". Fermat was part of the first generation who could claim to have mastered most of ancient math and to have invented new ideas. Previous scholars such as Fibonacci made improvements in notation and added some small novelties, but these achievements were trivial compared to the ancient knowledge which had been lost.

For example, today it is generally believed that Euclid wrote about Quadric Surfaces (in a text called "Surface Loci") - a field that Fermat was the first modern to even touch upon. Fermat's contemporaries struggled to understand Pappus's work on Projective Geometry - a very hot topic in math at the time - and were painfully aware that many of Pappus' works were completely lost.

Fermat's statement is entirely appropriate. He deserved to be proud of his accomplishment, exceeding the ancients in one small way, and he properly expressed the humiliating reality. It would take at least one more generation of mathematicians as brilliant as Fermat before the moderns could claim to have exceeded the ancients.

I am extremely interested in the idea that there was a period in time when, at least in the west, people looked upon the ancients (Romans/Greeks) as possessors of wisdom that had been lost.

In the light of this question its worth looking at the first volume of Richard Feynmans Lectures on Physics which contains the following quote:

If, in some cataclysm, all scientific knowledge were to be destroyed, and only one sentence passed on to the next generation of creatures, what statement would contain the most information in the fewest words?

I believe it is the atomic hypothesis (or atomic fact, or whatever you wish to call it) that all things are made of atoms—little particles that move around in perpetual motion, attracting each other when they are a little distance apart, but repelling upon being squeezed into one another. In that one sentence you will see an enormous amount of information about the world, if just a little imagination and thinking are applied.

Interestingly, given the thought experiment that Feynman posits this was exactly the situation that the atomic hypothesis, a notion from antiquity, faced having been 'lost' from Europe in late antiquity. It arguably helped resuscitate the scientific tradition in Europe, though the IEP says

It is probably an exaggeration to say that the restoration and study of Lucretius' poem was crucial to the rise of Renaissance "new philosophy" and the birth of modern science. On the other hand, one must not ignore its importance as a spur to innovative sixteenth- and seventeenth-century scientific thought and cosmological speculation.

But given Feynmans thought experiment it seems he considered it as crucial to the rise of physics as a scientific discipline, and personally I'd go along with that judgement. It's also worth noting that the reality of atoms, as such, wasn't generally established until 1905, when Einstein published a paper quantifying Brownian motion via the atomic hypothesis, whilst at the same time noting that QM demolished the naive conception of atoms in favour of a kind of synthesis of becoming and being that Heisenberg, in his later thinking of QM, called potentiality and actuality, terms he directly borrowed from Aristotle.

Its also important, given the slant of the question, to look at the term 'renaissance', which is predicated that it was the fresh look at Europes classical heritage at the time as being a precursor to the 'new philosophy' which lead to the scientific revolution. Certainly, Liebniz, the joint inventor of the calculus along with Newton, felt so when he advocated looking at the term entelechy in a paper which he wrote on the analysis of motion.

You also say:

Fermat near the end of his life told someone that he had done math to show that "the Ancients did not know everything" (not sure of exactly what he said).

Even mathematical greats aren't free of mathematical forms of superstition, which in its debased forms, like numerology are akin to astrology. One doesn't need to look at 'maths' to see 'that the ancients didn't know everything'. The question ought to be whether fresh insights can be found elsewhere, and this includes the past as well as other countries such as China (gunpowder and the compass), Egypt (monumental building) or India (Arabic numerals) and Arabia (optics).

I am sure that Fermat was serious. Sciences, as we know them, especially mathematics, astronomy and physics were invented in Ancient Greece and then reached a high level of development in the Hellenistic states. After the collapse of the Roman empire, this knowledge was practically forgotten, and there was almost no development. This was a true Dark Age, at least in what concerns science. (No matter what some post-modern historians say). So when the Renaissance of the antique knowledge began, people were under the strong impression that the ancients "knew everything", and that the real source of knowledge was careful study of ancient authors. This only started to change when Fermat worked, that is in 17th century. People like Galileo and Bacon had to prove that experiment is a more reliable source of knowledge than the writings of Aristotle. And Fermat started to discover things about numbers which the ancients did not know. Just think, what discoveries were made in number theory in the time after Diophantus and Fermat. You can hardly name anything. (The situation was somewhat better in the world of Islam and possibly in India and China, but how could Fermat know about this). This attitude was common in 16th and 17th centuries literature. Similar attitude prevailed in arts by the way.

Others have covered Fermat's note specifically, so I'll respond to this sub-question:

I am extremely interested in the idea that there was a period in time when, at least in the west, people looked upon the ancients (Romans/Greeks) as possessors of wisdom that had been lost.

After the fall of the Roman empire (around 476 AD - exact opinions vary at what point it "fell"), western civilization basically collapsed, and nearly everyone was forced back into subsistence living (i.e. barely having enough food to survive) for several generations. When in subsistence living, there isn't enough excess food to support full-time scientific jobs like teachers and scholars, especially when also having to deal with raids from foreign Germanic (and later viking) tribes.

As such, literacy plummeted and knowledge non-essential to farming and survival mostly evaporated.

This is what people (wrongly) call "the Dark Ages" and (also wrongly) blame on Christianity for 'persecuting scholars' (in actuality, Christianity preserved a lot of documents, and mostly supported scholars (with some notable exceptions in various places and times), and started schools, hospitals, and universities. As you noted, plenty of scientific progress was still made during this period).

The actual cause of the (now rejected by historians) ages being "dark" was the several generations of illiteracy due to the civilization collapse that basically "reset" western scientific knowledge, forcing scientific truths to be rediscovered gradually. Further, the lack of central authorities made travel more dangerous, slowing the sharing of knowledge. Scientific progress usually comes by one person having an initial seed of an idea, and several others in other locations seeing the significance of it - when communication between scientists in hampered, science itself is hampered.

Word-of-mouth oral history of the grandeur of the Roman empire persisted through the ages, so people were aware of it, as well as capable of seeing the Roman ruins in Italy and around Europe as a whole (including in Britain), so Roman achievements and capabilities were definitely admired and worshiped (in a non-religious sense) by scholars and poets.

Eventually, ancient Roman/Greek scientific books from the Eastern half of the old Roman empire (which had never collapsed to subsistence living in the first place), now under Islamic control, and formerly centered in Constantinople/Istanbul, made their way back to the West and greatly aided Western recovery of, and surpassing, the knowledge of the Greeks and Romans. These books confirmed the wisdom of the ancients, and added to their superhuman status, basically proving the myths about their superior civilization.

However, only some of the Greek and Roman scientific books survived through the intervening centuries - even in the East, which never 'collapsed' per se but endured repeated wars, lootings, and destruction of cities (Constantinople alone was laid siege to over 20 times; most failing, but still taking a serious toll on the surrounding regions). It was these lootings and destructions that actually drove or carried some of the books back into the West for the first time in centuries.

So, the myth persisted (partially correct) that the Greeks and Romans had immense amounts of knowledge lost after the collapse, and that (mostly incorrect) any new scientific discovery was just rediscovering something the ancients already knew, especially in the "purer" branches of sciences or philosophy and the various fields of math (especially geometry).

(Because many of the ancient works that survived through the ages reference other works that remain undiscovered, we (and the scholars of Fermat's time) get a glimpse at how much was lost - thousands and thousands of works, and those are only what we know about).

There is a difference between abstract knowledge and "inventions". In the 17th century it was still widely believed that the ancient Greeks had discovered and formulated pretty much the sum total of abstract knowledge. Fermat put this in question. The authentic quote from Fermat (in French and in English translation) can be found here:

Perhaps, posterity will thank me for having shown that the ancients did not know everything.