Why did Britain, not the Netherlands, create the greatest overseas empire?

The Dutch had three powerful enemies

1. The British at Sea- speaking generally, the Brits took more Dutch ships than the Dutch took British ships and British envy of Dutch prosperity played a large part in their decline.

2. France on land- fortunately, the Brits weren't keen on France being able to dominate the Channel by gaining Holland itself. Still, France was the greatest military and economic power of the time and so the Dutch were bound to decline. William III saw that a Protestant England could become a friend to Holland and thus supported his wife's claim to the Throne but he did not invade till invited to do so by prominent Englishmen. Once Churchill defected to William, James II knew his position was hopeless. Ironically, it was the fact that he now had a son and heir which made his position untenable. The English saw they had to get rid of a Catholic dynasty if they wished to maintain their Protestant Religion.

3. The Dutch were divided and, at times, their own worst enemy. However, had they- like the English- been an island and thus safer from overland invasion, their prosperity may have continued because, truth be told, they were and are a remarkable people

Several reasons made it less likely the Dutch ruled the seas:

1- Location The UK lies strategically 'in the way' for Dutch maritime expansion. From any direction, the Royal Navy had a strategic advantage. They could close traffic from the Americas and the Indies by patrolling the Channel. The more important Baltic traffic could also be blocked by the British.

Another reason was the shallow depth on the Dutch side of the North Sea. Dutch naval vessels were generally shallower draft and smaller than British vessels.

2- Lack of manpower The Dutch navy did employ many foreign sailors, but generally speaking, a population of <2 million need some significant advantages to win attritional wars against a country with at least 10 times as many inhabitants.

3- Lack of interest The Dutch empire was a commercial empire. Very few permanent colonies were founded by the Dutch. They founded many factories (warehouses, maritime outposts) that would be closed down when no longer profitable.

I happen to live in Thailand. Baan Hollanda is a museum, build on the old Dutch commercial settlement in Ayutthaya. That settlement was closed and reopened several times, pending the commercial and military situation.

Mass migration to any colony (Nieuw Amsterdam, Recieve, Kaapstad, Batavia) rarely happened. Certainly not on the scale the British practised it.

4- Conflicting interests If only the UK was the Dutch enemy, matters might (not likely) be different. However, France was also an enemy. The maritime provinces (Holland, Zeeland, Frisia) wanted a strong navy. The land provinces (Gelderland, Groningen, Utrecht and the prince of Orange) wanted a strong army. That was too much, the Dutch could afford a strong navy or a strong army but not both. That was a serious problem, usually the maritime provinces prevailed.

Another problem was internal division. Several internal civil wars, serious religious conflicts and several coups d' etat took place in the republic. (Maurits vs van Oldebarneveld, Gomaristen vs Arminianen, Willem II).

The main reason why the Dutch briefly were a world power were technical advantages (the Dutch build better and more economical ships) and civil wars in France and England. Once those civil wars were over, the Dutch position steadily declined.

There were several major advantages in favor of the British.

1. Better position. The British had a better western facing on the Atlantic. To get there, the Dutch had to navigate the North Sea or the English Channel, both of which were controlled by the British. The Dutch had the advantage in "eastern" (e.g. Baltic trade), but that was limited consolation.

2. Sheer size. The (future) United Kingdom has a land area of 242,000 square kilometers, versus 42,000 for the Netherlands, a factor of 6 times. The British population of 9 million (in 1700) was five times that of the Netherlands' 1.8 million. Britain's population was smaller than that of Spain, France, Russia or the Holy Roman Empire, but larger than that of Poland, Portugal and Venice (all of which were ahead of the Dutch). Britain definitely had critical mass, the Dutch, perhaps not.

3. Head start. British colonization of North America began with Henry Hudson before 1610. (And that's if you don't count Francis Drake's depredations in the 1570s and 1580s)*The Netherlands wasn't a fully independent country until 1648 (although its colonization started before that).

*A study by John Maynard Keynes showed that the difference in British growth trajectories before and after 1585 was directly attributable to the gold that Drake took from the Spanish.

I believe the main reason for the decline of the Dutch Republic was certainly the combined rivalry with both France and Britain, but mainly it is owed to its own economic decline around 1730, its own mentality when pursuing colonial ventures and the rise of its enemies.

At the offset of the first three Anglo-Dutch wars the Republic's naval supremacy was undisputed. From the Wikipedia page on the Anglo-Dutch wars:

By the middle of the 17th century, the Dutch joined the Portuguese as the main European traders in Asia. This coincided with the enormous growth of the Dutch merchant fleet, made possible by the cheap mass production of the fluyt sailing ship types. Soon the Dutch had one of Europe's largest mercantile fleet, with more merchant ships than all other nations combined, and possessed a dominant position in the baltic trade.

These wars were definitely hazardrous for the Dutch and caused losses in their trading business. As mentioned here,

...the English seized over 2000 Dutch ships, while the Dutch seized only 500 English ships.

Additionally, the English Naviagtion Acts that demanded imports be done only by English vessels from the exported countries failed to achieve their goal of limiting Dutch trade. Specifically as mentioned by C. H. Wilson in his work The Economic Decline of the Netherlands:

The English Navigation Acts were riddled by exemptions on licence, as well as by the clauses in the Treaty of Breda which nominated the Rhineland as Holland's rightful hinterland, whose goods counted as her own.

The major Franco-Dutch war betwenn 1672-1678 also had significant economic impact for the Dutch, as wars often do, but it was by no means a decisive blow for the Republic. Specifically by late 1672:

In June, the Dutch seemed defeated. The Amsterdam stock market collapsed and their international credit evaporated... Instead of a rapid victory, Louis was forced into another war of attrition around the French frontiers; in August, Turenne ended his offensive against the Dutch and proceeded to Germany with 25,000 infantry and 18,000 cavalry.

Aditionally, the Dutch, chartered a West Indies Company (WIC) that had in its duties the handling of colonization in the Western Hemisphere.

“[…] The Company [red. WIC] was “empowered to do everything that the service of this country and the profit and the increase of the trade shall require,” from the pacific colonization of fertile and uninhabited places to open warfare if a powerful and continued obstruction of the Company`s trade and navigation so required.”

In this study by Jesper Nicolai Abel named Early Dutch presence in North America, the claim is stated that Dutch preffered trade over agriculture when searching for wealth which is more easilty understood that by the fact that merchants were the ruling class in the Republic at the time. Thoughts were made by the WIC to emphasize in the creation of a larger colony, based on agriculture that would encorporate all the trading posts founded for fur trade in North America but these were ultimately rejected. Willem Usselincx was a major supporter of the further colonization of the Americas and feared that the new WIC company would focus on profit rather than colonization.

This large-scale colonization proclaimed by Usselinx required a considerable emigration, and it remained to be seen whether merchants would be eager to undertake the risks inherent to such ventures. However, the States-General and the Heren XIX remained unconvinced by the economical aspect of Usselinx`s plan and it had no place in the “Grand Design” (Groot Desseyn), which was drawn up by the Lord Nineteen in October 1623.1

But in the end, the Dutch were caught up by the times and their pioneering practices that helped give them a competitive edge were eclipsed as supported here and here:

The slow economic decline after 1730 was relative: other countries grew faster, eroding the Dutch lead and surpassing it. Wilson identifies three causes. Holland lost its world dominance in trade as competitors emerged and copied its practices, built their own ships and ports, and traded on their own account directly without going through Dutch intermediaries. Second, there was no growth in manufacturing, due perhaps to a weaker sense of industrial entrepreneurship and to the high wage scale. Third the wealthy turned their investments to foreign loans. This helped jump-start other nations and provided the Dutch with a steady income from collecting interest, but leaving them with few domestic sectors with a potential for rapid growth.

Specifically, in a study of the Anglo-Dutch Economic Relations in the Atlantic World, 1688–1783, the following important fact is stated:

The Dutch played a significant role in establishing sound, effective public credit in Britain after the Glorious Revolution. Dutch financiers and advisors accompanying William of Orange to London brought with them practices such as the resale of shares in joint-stock corporations. This helped to establish stock exchanges in Britain. The Bank of England was modeled on the Amsterdam Wisselbank; the national debt also followed Dutch practice. Adam Smith noted the importance of Dutch overseas investment: “The mercantile capital of Holland is so great that it is…continually overflowing, sometimes into the public funds of foreign countries.” Dutch investors were attracted to British securities by the high return on English bonds and by the convergence of Dutch and English public debt institutions.

The ons and offs of Anglo-Dutch relations were not enough to stave of trade competition far from home, as stated in Anglo-Dutch Connections and Overseas Enterprises: A Global Perspective on Lion Gardiner's World:

For all their common struggles in Europe, the English and Dutch were hardly automatic allies when far from Home: the Dutch friend could also be a foe, and nowhere was the competition more heated than in the trading posts of the East Indies.

Later, the inauguration of William of Orange as King of England had mixed results for the Dutch:

The Glorious Revolution of 1688 ended the 17th century conflict by placing William of Orange on the English throne as co-ruler with his wife Mary. This proved to be a pyrrhic victory for the Dutch cause. William's main concern had been getting the English on the same side as the Dutch in their competition against France. After becoming King of England, he granted many privileges to the Royal Navy in order to ensure their loyalty and cooperation. William ordered that any Anglo-Dutch fleet be under English command, with the Dutch navy having 60% of the strength of the English.

The creation of closer ties between English and Dutch interests was not enough to cool down the trading competition as shown in the history of the Dutch East India Company, officially the United East India Company (Dutch: Vereenigde Oostindische Compagnie; VOC):

In 1710, the Zamorin was made to sign a treaty with the VOC undertaking to trade exclusively with the VOC and expel other European traders. For a brief time, this appeared to improve the company's prospects. However, in 1715, with EIC encouragement, the Zamorin renounced the treaty. Though a Dutch army managed to suppress this insurrection temporarily, the Zamorin continued to trade with the English and the French, which led to an appreciable upsurge in English and French traffic. The VOC decided in 1721 that it was no longer worth the trouble to try to dominate the Malabar pepper and spice trade. A strategic decision was taken to scale down the Dutch military presence and in effect yield the area to EIC influence.

Especially the first reason, was of paramount importance to the Dutch naval decline. The original Dutch naval power was based on their large mercantile fleet that transported goods all over Europe. As their enemies started investing in their own naval forces, England doing so as a direct result of their rivalry with the Netherlands, Dutch ships became less relevant. For example the following extract from the Wikipedia page on the Flyut ship is enlightening:

This ship class was credited in enhancing Dutch competitiveness in international trade, and was widely employed by the Dutch East India Company in the 17th and 18th centuries. However, its usefulness caused the fluyt to gain such popularity that similar designs were soon developed by seagoing competitors of the Dutch. For example, the English shipbuilding industry began to adapt the design of the fluyt during the later part of the 17th century as English merchants, seeing how much cheaper the Dutch shipping was, acquired Dutch-built ships that were captured in the Anglo-Dutch wars.

Therefore, toconclude, the need for naval investment required to support a large overseas empire, was gradually eclipsed for the Dutch, because firstly large scale colonization was never that appealing to them and secondly they found ways to make money through funding their former rivals' empires. The long 17th century antagonsim with two major rivals, England and France, tooks its toll and in the end the Dutch strategically shifted their economic activities.