Why didn't colonial empires exchange their colonies with others empire to make their territories whole?

Scattered Holdings were Helpful

Along with Mark's excellent answer, there is another reason why they wouldn't trade to get contiguous colonial holdings. Ports. A ship can only be out and doing for so long. Eventually food runs low, hulls get covered in barnacles, spars snap, coal needs to be replenished. When that happens, a ship needs to head in to a friendly port. Even more than that, a ship that just came out of port, crew rested, fully stocked, is more capable than one that's been on the ocean of weeks or months at a time. You can even keep ships stationed in far-flung ports so that if trouble starts you have ships close to the action! Otherwise it might be weeks or even months for ships to go from their peacetime stations to where the action is.

It wasn't until WWII that underway replenishment was really worked out, and only then by the US Navy, the Japanese were playing defence and the Brits had tons of... you guessed it... scattered colonial holdings. Those little bits of islands/coastline that look random and less valuable than a concentration of power actually allowed the Royal Navy to operate anywhere in the world with relative ease.

They did. Sometimes.

In the Heligoland treaty, Germany not just swapped Zanzibar for the little island in the North Sea but also adjusted other territories with the British Empire.

The shorthand answer is that the age of colonial empires was not a board game where each empire has a a strategy, and there's a way of scoring a winner. Empires -- like everything human -- were complicated with multiple (often conflicting) motivations and multiple (often conflicting) views on how international relations would evolve.

Doubtless some colonies were seized just to add acreage to the Empire, but there were so many other reasons:

• Raw materials (the spice islands, gold, etc.)
• Coaling stations to facilitate haulage between home and important sources of wealth
• Defensive: It's a great place to dominate a narrow sea (e.g., Gibraltar)
• Winnings: After a war (e.g., the Seven Years' War), the loser frequently had to pony up some colonial possessions to the victor.
• Ma, it followed me home, can I keep it? The British in particular seemed to keep picking up territory when some entrepreneur went off and conquered something for personal gain and added to the Empire
• Pure accident. Captain Whatsit sailed by, discovered an island, claimed it for Home, and as it was (a) distant and (b) worthless, no one contested it.

This is not how a tidy Empire with defensive borders is created! Why didn't they regularize the borders?

First, in each colony, there were people on the ground who owned property there. No matter how much geopolitical sense it made to trade a chunk of Existan for the Island of Whatsit, the rich colonials who owned chunks of Existan would object. "What? You mean to give up the part of our Sacred Empire? Don't you care? If you really want Whatsit, go conquer it like a civilized person."

Second, until fairly late in the colonial game, the borders were where they were because they didn't matter that much. The colonies were centered around rich territories of strategic locations, and the hinterlands were full of natives, if that. Bordering rivals were in much the same position. The small armies in the colonies could not seriously threaten the other Empire (which could always bring in bigger guns), and Empire vs. Empire conflict would be in Europe or on the high seas.

Third, many of the colonies existed for strategic reasons and needed to be right where they were.

Bottom line: Multiple reasons for having a colony plus multiple conflicting private interests equals a confused muddle.