What is the significance of 4200 BCE in context of farming replacing foraging in Europe?

It feels as if the question draws a frame around the problem that is a bit misleading.

The quote in question needs a bit more context:

Neither [is optimal, as, LLC] all the theories treat the triumph of farming as inevitable. Competition, genetics and archaeology imply, has little to do with exams or teachers, because it has always been with us. Its logic means that things had to turn out more or less as they did.

But is this true? People, after all, have free will. Sloth, greed, and fear may be the motors of history, but each of us gets to choose among them. If three-quarters or more of Europe’s first farmers descended from aboriginal foragers, surely prehistoric Europeans could have stopped farming in its tracks if enough of them had decided against intensifying cultivation. So why did that not happen?

Sometimes it did. After sweeping from what is now Poland to the Paris Basin in a couple of hundred years before 5200 BCE, the wave of agricultural advance ground to a halt.

^{Figure 2.4. Going forth and multiplying, version one: the westward spread of domesticated plants from the Hilly Flanks to the Atlantic, 9000–4000 BCE}

For a thousand years hardly any farmers invaded the last fifty or sixty miles separating them from the Baltic Sea and few Baltic foragers took up more intensive cultivation. Here foragers fought for their way of life. Along the farming/foraging fault line we find remarkable numbers of fortified settlements and skeletons of young men killed by blunt-instrument traumas on the front and left sides of their skulls—just what we would expect if they died fighting face-to-face with right-handed opponents using stone axes. Several mass graves may even be grisly relics of massacres.

We will never know what acts of heroism and savagery went on along the edge of the North European Plain seven thousand years ago, but geography and economics probably did as much as culture and violence to fix the farming/foraging frontier. Baltic foragers lived in a chilly Garden of Eden, where rich marine resources supported dense populations in year-round villages. Archaeologists have unearthed great mounds of seashells, leftovers from feasts, which piled up around the hamlets. Nature’s bounty apparently allowed the foragers to have their cake (or shellfish) and eat it: there were enough foragers to stand up to farmers but not so many that they had to shift toward farming to feed themselves. At the same time, farmers found that the plants and animals that had originally been domesticated in the Hilly Flanks fared less well this far north.

We frankly do not know why farming did finally move north after 4200 BCE. Some archaeologists emphasize push factors, proposing that farmers multiplied to the point that they steamrollered all opposition; others stress pull factors, proposing that a crisis within forager society opened the north to invasion. But however it ended, the Baltic exception seems to prove the rule that once farming appeared in the Hilly Flanks the original affluent society could not survive.

In saying this I am not denying the reality of free will. That would be foolish, although plenty of people have succumbed to the temptation. The great Leo Tolstoy, for instance, closed his novel War and Peace with an odd excursus denying free will in history—odd, because the book is studded with agonized decisions (and indecisions), abrupt changes of mind, and not a few foolish blunders, often with momentous consequences. All the same, said Tolstoy, “Free will is for history only an expression connoting what we do not know about the laws of human history.” He continued:

The recognition of man’s free will as something capable of influencing historical events … is the same for history as the recognition of a free force moving the heavenly bodies would be for astronomy … If there is even a single body moving freely, then the laws of Kepler and Newton are negated and no conception of the movement of the heavenly bodies any longer exists. If any single action is due to free will, then not a single historical law can exist, nor any conception of historical events.

This is nonsense. High-level nonsense, to be sure, but nonsense all the same. On any given day any prehistoric forager could have decided not to intensify, and any farmer could have walked away from his fields or her grindstone to gather nuts or hunt deer. Some surely did, with immense consequences for their own lives. But in the long run it did not matter, because the competition for resources meant that people who kept farming, or farmed even harder, captured more energy than those who did not. Farmers kept feeding more children and livestock, clearing more fields, and stacking the odds still further against foragers. In the right circumstances, like those prevailing around the Baltic Sea in 5200 BCE, farming’s expansion slowed to a crawl. But such circumstances could not last forever.
_{–– Ian Morris: "Why the West Rules--for Now: The Patterns of History, and What They Reveal About the Future", 2011.}

That means we do have a process to observe in the archaelogical evidence: ranging from 5200 to 4200 BCE. The Linearbandkeramik culture brought their way of life into Poland around 5200, then tried to expand into the Baltic, but the locals wouldn't have it. And they were numerous and well fed and warring enough to stay. For a while, of around 1000 years.

A phenomenon that appeared repeatedly:

In the northern half of Europe, the potential of the new system of cultivation was immense. […] The new areas included forests and moors that existed either on permeable and leached soils that were not fertile for cultivation without manure or on soils that were too heavy to be cultivated without the plow. Other areas were coastal marshes, freshwater marshes, and wetlands in the interior that were difficult to drain and cultivate without heavy equipment. Finally, there were particularly cold regions […] and the northern areas of Scandinavia, Poland, and the Baltic countries. All these regions were thus relatively or totally uninhabited. Moreover, they were called “deserts” even if hunters, slash-and-burn farmers, shepherds, fugitives, and brigands were sometimes encountered there. These were relatively insecure regions, and roads suitable for wheeled vehicles often made large detours in order to avoid them.
_{–– Marcel Mazoyer & Laurence Roudart: "A History of World Agriculture: From the Neolithic to the Current Crisis", Earthscan Publications: London, Sterling, 2006. Obviously about the Middle Ages, p286/7. (PDF)}

After this culture we have to observe the Pitted Ware culture that partially morphed into the Funnelbeaker culture.

For the pitted ware especially it seems that a cultural convergence with neighbouring cultures, farmers, took place. So much so that in the corded ware we cannot make distinctions any more with our evidence we dig up.

Further we need to observe that regardless of any 'warm' or 'cold' period: that edge of the Baltic Sea is just always "colder" than Southern France. And since the Hilly Flanks crops need to be adapted by breeding, to colder climates, we might speculate quite reasonably that for expanding by conquering farmers they reached a zone of diminishing returns for their methods of cultivation, while the locals could sustain themselves by traditional, mesolithic methods, despite taking a liking in some of the handy tools the newcomers had with them.

The date of "4200 BCE in the Baltic" is therefore of as low significance as our knowledge of "what happened in that year". We simply see a change in archaeological evidence around that time in that region. What 'event', if any one such appeared, will remain nebulous to us, as the shards and bones don't tell us enough.

Almost the same 'pattern' we see with the Ertebølle culture 5300 BC – 3950 BC much further West than the Baltics. A late-meso-lithic ceramics producing culture that continues hunting and gathering and was in contact to farming societies to the South.

The two case studies presented here lead us to reconsider the conventional definitions of 'farmer' and 'forager'. These have largely been based on the belief that the first signs of farming in a given area constitute a stadial transformation along the lines of a 'neolithic revolution'. This in turn springs from the assumption that 'neolithic' social and economic forms are necessarily an advance over 'mesolithic' ones, and that the first traces of a neolithic economy, however dubious, constitute some giant leap for mankind.
Our model suggests, on the other hand, that any major changes should be looked for at a later stage of the process, during the substitution phase. Major changes in settlement pattern, the organization of labour, of society, even of symbolic expression etc., are unlikely to occur until agricultural practices become predominant—however long this takes.
[…]
Both the case studies emphasize the long continuation of foraging adaptations, and the long delay before the appearance of predominantly agricultural economy. For both Denmark and Finland we believe that the cause of this was the existence of successful maritimeadaptations. Fishing communities, because of their normally greater group size and decreased mobility, have often been regarded as pre-adapted to make the change to agriculture quicker than other groups. This is based on (a) the anthropological practice of placing present-day societies into a. typological sequence and assuming that this is also a development sequence, and (b) the assumption (until recently generally held) that fanning was necessarily a superior adaptation. By virtue of their more similar life style, fishers could therefore become farmers quicker and easier than could other foragers.
This seems to be an area in which archaeology can actually amend anthropologically-derived theories. The two cases discussed here suggest that the opposite is the case: the maritime adaptations remained for long periods viable alternatives to farming, and were thus in themselves the very reason for the delay in the spread of farming. This would have been particularly the case during the early stages of agricultural availability—agriculture on its northern margin would inevitably be ecologically less suited than elsewhere, and so relatively unattractive until new strains and techniques could be developed. In both Denmark and Finland a specific trigger (a decline in marine resources) was necessary to initiate the substitution phase. Work in Japan has produced a similar result: agriculture did not replace maritime Jomon groups in eastern and northern Honshu until a decline in marine productivity occurred (Akazawa 1981). The degree of compatibility between foraging and farming varies between areas. If swidden cultivation is employed, competition between the two is to some extent mitigated (see the Finnish case, above); this may be one reason why the substitution phase lasted so much longer in Finland than in Denmark.
In general, however, we believe that once any major shift to agriculture had begun, its effects would be so disruptive for foraging that increased reliance on agriculture would be inevitable. While the initial adoption of the elements of farming might have taken place for a variety of reasons, the subsequent outcome of this process was bound to result in the demise of the foraging economy and in full transition to farming. If this is a general pattern, the implications are clear: far from being adopted for its social and economic benefits, the neolithic economy was in the end adopted because of lack of alternative strategies which would preserve the hunting and gathering way of life once the transition got underway.
_{–– Marek Zvelebil & Peter Rowley‐Conwy: "Transition to farming in Northern Europe: A hunter‐gatherer perspective", Norwegian Archaeological Review, 17:2, 104–128, DOI}

A narrative that paints a picture of a continuous and never stopping 'coz unstoppable' advance on an unbroken frontline seems correct –– only if we look at very long timeframes, and from very far above. But the situation on the ground wa probably a lot more patchy than the first map in this answer and the Morris quote might suggest:

WP: European Middle-Neolithic, (from Linear Pottery culture)

The dateline of "4200" is a shorthand that leaves out all the uncertainties and ranges around this epoch-marker for the region in question. It is much less precise and much less definite than we would like to infer from the one sentence in the book.

Farming period—hunter–gatherers in the farming world and their role in core-periphery systems
Farming was introduced from Central Europe between 6400–6000 BP (5300–500 BC, 4400 and 4000 bc) into northern Poland and Germany by enclave forming, isolated settlements of the LBK and derivative (SBK, Lengyel) traditions. Following this episode, the Wrst extensive farming communities in northern Poland and Germany, Denmark, southern Norway and southern and middle Sweden belong to the TRB culture and date from ca. 5700 BP (4600 BC, 3700 bc) on the north European Plain, and from ca. 5200 BP (ca. 3200 bc, 3900 BC) in southern Scandinavia (Bogucki, 1996, 1998, 2000; Fischer, 2003; Fischer and Kristiansen, 2002; Midgley, 1992; Nowak, 2001; Price, 2000). In northern Scandinavia and more eastern regions of the Baltic the agricultural transition unfolded between 4500 and 2500 BP (2500–500 bc, Antanaitis, 2001; Antanaitis et al., 2000; Daugnora and Girinkas, 1995; Zvelebil, 1981, 1987, 1993).
At the same time, in certain regions, such as Silesia, Kashubia, Mazovia, and Masuria in Poland, hunter–gatherer communities survived into the Bronze Age (until ca. 1500 bc, Bagnienvski, 1986; Cyrek et al., 1986; Kobusiewicz and Kabacijski, 1998), in parts of Lithuania hunter–gatherers continued until ca. 500 bc (Antanaitis, 2001; Daugnora and Girinkas, 1995). In southern Finland, farming was gradually adopted between 3500 and 2000 bp (1500 bc—0, Meinander, 1984; Taavitsainen et al., 1994, 1998; Vuorela, 1976, 1998; Vuorela and Lempiäinen, 1988; Zvelebil, 1981). In Swedish Norrland, and in northern and eastern Finland the transition only ended in the 16th–17th centuries AD, in early modern times with the domestication of reindeer by the Saami and the development of swidden farming among the Karelians (Mulk and Bayliss Smith, 1999; Orrman, 1991; Taavitsainen et al., 1998). In this sense, there is no break between the Mesolithic hunter–gatherer communities of the early post-glacial period and the later prehistoric and early historical hunters within this region. Rather than viewing these later hunter–gatherers as stone age survivals, however, we should regard them as communities who have successfully responded to the historical necessity of living in an increasingly farming world by developing the trading potential of hunter–gatherer existence: they became commercial hunter–gatherers.
_{–– Marek Zvelebil: "Mobility, contact, and exchange in the Baltic Sea basin 6000–2000 BC", Journal of Anthropological Archaeology 25 (2006) 178–192.}

Adding to the above factors, a possible influence of either prevailing or changing climate and necessary adaptions in people, crops or methods might be illustrated with another map. The generalised assumption of farmers always outperforming hunters on the same area obviously relies on thge assumption of 'areas' being the same or at least quite comparable. The region in question offers these differences concerning area:

_{Plant & Soil Sciences eLibrary: Soil Genesis and Development, Lesson 6 - Global Soil Resources and Distribution}

For the North-Eastern farming frontier lasting so long:

In addition to the small number of sites and numerous isolated finds, pollen diagrams of lake and bog sediments may reveal evidence of Early Bronze Age settlement areas. Clear signs of human activity reshaping the environment were already present at the beginning of the Neolithic. At several locations this change is also correlated with the first sporadic evidence of cereal cultivation (Veski 1998; Kriiska 2003, tab. 1), whereas farming indi- cators became stronger only at the end of the period, in c. 2200–2000 BC (Veski & Lang 1996a–b; Saarse et al. 1999). A significant increase in human impacts during the Late Neolithic was followed by a subsequent decline at the beginning of the Bronze Age. The periods of decline in activity were dated differently in different areas and they were followed by a new increase either at the end of the Early Bronze Age or at the beginning of the Late Bronze Age (see e.g. Saarse et al. 1999; Veski & Lang 1996b; Pirrus & Rõuk 1988; Poska et al 2004; Laul & Kihno 1999; Kihno & Valk 1999).

Thus, the character and extent of human impacts differed in various regions and times. An important characteristic is that the periods of major human impact were rather short and were replaced by periods of decline; decrease in human impact in some places was followed by a rise in other regions. As the pollen diagrams reflect the environmental changes only in the vicinity of sampling sites, the situation seems to indicate considera- ble instability, at least in regard to the location and use of arable land; settlements in general were likely impermanent. One can assume that people continued to look for better and more suitable places for farming. The character of settlement and economy, and the respective reflections in pollen diagrams, were basically the same in south- western Finland, Latvia, and Lithuania at the time (see Lang 1999:367-368).

The first landnam and the transition from hunting and fishing to farming in general was a remarkably long process. It took 2500 years in northern and western Estonia and up to 3500 in central and southern Estonia from the emergence of Cerealia pollen in the dia- grams until the establishment of societies where the main means of subsistence was agriculture. The situation is similar in other countries on the eastern coast of the Baltic Sea; farming societies were established in Lithuania, Latvia, and Finland during the Late Bronze Age at the earliest, and even then not in all regions (e.g. Antanaitis 2001; Anta- naitis-Jacobs & Girininkas 2002; Zvelebil 1993). It is commonly held that such a slow transition to farming can be explained by unfavourable climate and the plentitude and availability of alternative resources for hunting and fishing. Both explanations are ob- viously valid to a certain extent, but they are insufficient for understanding the whole process.

The long and gradual transition period provides indirect evidence that the process in- volved local populations, not in-migration of farming tribes. The fact that the develop- ment of a new dispersed settlement pattern was accompanied by the abandonment of the old settlement centres of hunters-fisher-gatherers, suggests that the occupants of the new areas were local, and supports the above claim (see Lang 2000b). On the other hand, the long transitional period involved a specific type of economy – complex fishing-hunting or the ‘Forest Neolithic’ economy (Zvelebil 1993, 157), which presumably fulfilled the subsistence needs of the small and dispersed population in the best possible way.

The transition to farming, which was a much more labour-intensive lifestyle than forag- ing (Sahlins 1974; Cashdan 1989) and yielded results after a longer period of labor (see Zvelebil 1993 and the literature cited), was not the consequence of economic difficulties (e.g. famine due to the lack of game, fish, seals) as generally thought before. Rather, the transition can be explained by the social needs and behaviour of the society at large, the significant factors here being social competition between the leaders of the society, trade of prestige items, and the manufacturing of grain-based alcoholic drinks to be consumed at (religious) celebrations and upon entering into various alliances (see e.g. Bender 1978; Sahlins 1974:149 pp.; Jennbert 1988). It can be assumed that the transition to farming, which was the best way to obtain additional resources, was more rapid and complete in regions where the social contacts both within and between the communities were closer and the competition between the leaders was more fierce because of the need to maintain such relations. The small size and low density of the population in Estonian and other areas on the eastern coast of the Baltic Sea during the third and second millennia BC explains why the above social needs and behavioural patterns did not develop here, at least not to the same extent as they did in the southern latitudes. It was the crucial absence of the social engine that determined a slower pace of economic growth.

_{–– Valter Lang: "The Early Bronze Age in Estonia: Sites, Finds, and the Transition to Farming", in: Helene Martinsson-Wallin (Eds): "Baltic Prehistoric Interactions and Transformations: The Neolithic to the Bronze Age", Gotland University Press 5 , Visby, 2010.}

The time period from roughly 7500 BP (years Before Present) to 4000 BP (5500 BCE to 2000 BCE), known as the Holocene Maximum (or Optimum) saw global temperatures:

• rapidly increase from slightly (~0.5°C) below current the present value to between 1 and 2°C higher;

• stay at those values for nearly 2000 years; and

• then return to values ~0.5°C below current.

(src: The climate of the Holocene)

As seen above, the peak of the Holocene Maximum is reached about 6500 BP (4500 BCE) – just when the disappearance of forager culture occurs in the Hilly Flanks.

Warmer global temperatures are accompanied by increased rainfall, longer growing season, and more consistent seasonal weather (due to less waviness in the Jet Stream): all favouring the spread of agriculture into the Hilly Flanks.

The great length of the Holocene Maximum, about 2000 years, ensured that forager cultures were completely replaced by agricultural ones before global temperatures fell again.

Note that the late Bronze Age collapse, often attributed to the unknown Sea Peoples, occurs about 3200 BP (1200 BCE), just as global temperatures approach their coldest value since, except for the early Modern Little Ice Age.

Farming societies typically support 60 to 100 times the population of hunter-gatherer societies. Given that kind of population difference, what that one person wants/needs vs. the 100 simply doesn't matter. They become unimportant on the ground, and are simply genetically and socially washed away in the tide. The hunter's options are to retreat to unfarmed land, learn farming and join everyone else, or try to stand their ground and get overwhelmed by superior numbers. No matter what, they cease to be an issue.

As for a mechanism, there are nothing but theories at the moment. I think the one in Pieter's answer is probably the one I'd lean toward for the overall timing of the Neolithic Revolution. Clearly humans worldwide were ready for it, because humans worldwide developed animal and plant husbandry independently at nearly the same time without any known contact with each other. It was probably only awaiting a climate where it was workable as a lifestyle.

As for the specific timing in Europe, it seems fairly clear (to me at least) that it's down to the emergence of the Indo-Europeans. Their protolanguage arose in western Asia at nearly exactly that time (up to 4500 BC). Worldwide they have a common DNA trait of adult lactose tolerance, so it seems likely this was part of the package, as was the milkable livestock who make that trait useful. When they reached the Balkans, they were able to mix that with borrowed near-eastern crops, and there's just no way scattered woodland native Old European hunters could stand against the population tide that followed.