Why were the populous coastal areas of China not linguistically united?

One reason was because China was "linguistically united," through its written language, which is basically common to most dialects of Chinese. The second reason is because transportation and communications in China, until the past 40-50 years was poor by modern standards; that is, it was more like medieval standards.

The written language was "enough" to connect China even though (until modern times) very few people traveled far from home. There was a small educated class (basically the literati) who traveled from one part of China to another, who could write to each other, and who could learn enough of each others' spoken language to communicate when necessary. The average person traveled and communicated with people within a radius of tens of miles, who spoke the same language. For these people, you are talking about transportation by walking, oxcart, horseback, or a few miles up or down river by "boat" or small ships. Until recently, few people had cars or access to trains and planes, and until the People's Republic was established, even bicycles were rare.

Much of Western Europe adopted some variation of Latin, because it was part of the Roman Empire. The whole empire (5 million square kilometers) is about half the size of China, and the part with Latin-based languages is one third of the Roman Empire (one sixth the size of China. That includes France, Italy, Spain, Portugal, and Romania. (To a certain extent, it was true of England, although the English language has strong admixtures of German elements.) Even though the languages have a common (Latin) base, issues of transportation and distance (before modern times) led to strong differences between the local "variations."

Despite appearances the “Han Chinese” don’t really conform to the stereotype of an ethnic group. The fact that they speak diverse local languages is just one aspect of this.

South-central China can be divided into a number of regional systems, each typically centered around the drainage basin of a major river, that do not correspond to administrative divisions. G. William Skinner pioneered the study of these macroregions, although we should note that he was primarily interested in economic and social relations, not linguistic ones.

In the introduction to an innovative study of the history of Hakka migration and ethnicity in the Lingnan region by Sow-Theng Leong, Skinner notes that speech group boundaries and regional ones line up:

For the most part, each of the major speech groups of southeastern China dominated one macroregional or subregional core. In the Lower Yangzi, it was Wu speakers of the Taihu subgroup. In the Ou-Ling subregion of the Southeast Coast macroregion, it was Wu speakers of the Oujiang subgroup. The core areas of the other three subregions of the Southeast Coast were each dominated by a distinct Min-speaking group: Hokchius in the Min basin, Hokkiens in the Zhang-Quan subregion, and Teochius in the Hanjiang basin. For the rest, Cantonese dominated the regional core of Lingnan, Gan speakers that of the Gan Yangzi, and Xiang speakers that of the Xiang basin subregion of the Middle Yangzi. Of all the major cultural/linguistic groups of South-Central China, only the Hakkas had no substantial drainage basin of their own. (p.3)

The Hakkas were the exception because their heartland was actually a mountainous area peripheral to several river systems. In the early Qing period, they migrated in large numbers to several different areas that were expanding economically, but by the 19th century changed economic conditions and overpopulation led to conflict between immigrant Hakkas and the core speech group (Cantonese in the Lingnan region, for example.) This conflict took on an ethnic character, so we could actually say that the speech groups within what is now called “the Han Chinese” behaved like ethnic groups in some ways. Yet according to Leong, there wasn’t much in the way of pan-Hakka solidarity across regions, even when the Hakkas in, say, Lingnan banded together for self-defense in the face of Cantonese discrimination.

So generally, linguistic/ethnic divisions within the “Han Chinese” can be explained by looking at regional systems within China.

Reference: Sow-Theng Leong, Migration and Ethnicity in Chinese History: Hakkas, Pengmin and their Neighbors (Stanford UP 1997)

Well, as other posters have noted, you're starting from some faulty premises.

Whilst most of China is quite culturally and ethnically diverse, there are significant areas especially on the coast and river plains which are mostly Han Chinese.

The image Semaphore linked shows the opposite of what he was trying to say. The areas of that map that you and he think look diverse (to the extent they've been able to resist incroaching Han settlement since it was drawn) are actually the monocultural backwaters. Most weren't part of China until the Qing dynasty and, because of low rainfall and unsuitability for Chinese agriculture, almost no one lives there. 96% of the 1.4 billion people in the PRC live east of the Hu Line drawn between SW Yunnan and central Heilongjiang.

As you can see, most of that maps 1:1 over the Han part of Semaphore's map. Even the majority of the 4% on the other side of the line live in western Shaanxi or Gansu's Hexi Corridor and are Han. On the populated side, the diversity you see in Yunnan and the NE on Semaphore's map is mostly illusory. Yunnan's mountains and the Qing's Manchurian policies kept the Han out for a while, but the former is now full of expressway tunnels and the latter's a long time gone; they really are diverse at the moment, but they're sinicizing rapidly and will soon only be distinguishable from the Han for tourism purposes.

On the other hand, like Neubau said, the idea that the Han themselves are a monolithic ethnicity is a political and cultural illusion. China is quite bigger than Europe sans Russia, and the Han part of China is still bigger than western Europe and Scandinavia.

Today most people under 40 can speak Mandarin because of the government education policies, but historically everyone but the government officials only spoke languages from the same linguistic family. Many of the southern 'Han' are the descendants of the Baiyue, Minyue, etc. who lived in the area before its conquest by the Zhou, Qin, and Han and their dialects continue to preserve some parts of their old languages. Prior to modernity, the cultures of northern and southern China were quite distinct and had completely different diets. Moreover, though it can be hard for Europeans to process, Chinese from different regions don't actually look the same. Using generalizations that are hopefully easier to visualize, northerners are taller and look more Mongolian, people around Shanghai closer to southwestern Japanese, and those in the south shorter and closer to the Vietnamese and Thais.

In other words, the 55 official minorities are increasingly joining a single Chinese culture but the Han themselves are quite culturually, genetically, and ethnically diverse. Pretending they aren't is like imagining that the French, Belgians, Spanish, Portuguese, and Italians are exactly the same ethnicity because they mostly have an easier time to talking to girls than the English, have dark hair, speak Romance languages, are Catholic, trade with each other, eat wheat-based carbs, and were formerly governed by Bourbon kings. It's even worse, actually, since those countries together make up 2½% of humanity.

The Han are 18% of all humans on Earth. The Belgians or Portuguese barely make up the population of a medium-sized Chinese city and have a much shorter history than most. Guangzhou and Beijing are about as far apart as Athens and Moscow or Valencia and Berlin.

These provinces are also united by a strong central government for much of their history, and did not spend any significant amount of time divided since the start of the Song Dynasty.

The Song very famously got pushed into the southern half of the country, the Yuan (who natively spoke Mongol) similarly held out in the north, the Ming regrouped to the south and even overthrew and colonized Dutch Taiwan, and the aftermath of overthrowing the Qing (who natively spoke Manchu) was anarchy, invasion, and civil war. So, no, it wasn't even notionally united for extended chunks of time even if you only focus on the Han parts of China.

Further, during the unified bits, imperial China wasn't ever strongly centralized by modern standards. It was better off than, say, Poland-Lithuania or early medieval France but, right up to the Xinhai Revolution, it ran its decentralized and often corrupt provinces as tax farms for the imperial court and national army. Standing policy was to never appoint officials to their own provinces because that would give them too much local support and risk rebellion, and (particularly under the Yuan and Qing) local factions were played off one another the same way the Brits ran India. Famously, even the Ming's Hongwu Emperor Zhu Yuanzhang viciously screwed over southerners to appease the north during his reign's first imperial exam.

Yet, they remained linguistically divided for centuries until the Communist Party declared that everyone should speak Putonghua/Guoyu/Mandarin, and even then local dialects are still very popular in the south.

First, local dialects were and are popular all over. (Again, 'dialect' here is mostly political and highly misleading. More people natively speak Wu dialects like Shanghainese than speak any kind of French.) The illusion you're describing is just an effect of the northern and central dialects being closer to the Beijingish dialect the Ming, Qing, and Commies used as their standard. Even Beijing's local dialect is famously not actually Mandarin but Mandarin as spoken by English pirates ("Be ye wanting some yangrouchuarrrh, me xiaonanharrrrh?"). Its dialect family was always more common in the North China Plain, and it took over Sichuan because of influxes of fairly recent refugees from the north. Even so, Nanjingese (which used to be standard in the south and is partially where you got all those weird k transliterations like kung fu for gongfu) isn't actually intelligible to someone from Beijing despite showing up as 'Mandarin' on linguistic maps.

Second, China's had unified national dialects as far back as the late Zhou, always based on what people spoke at the capital. So the first 'Mandarin' was the ancient version of Luoyangese; the prestige form of Middle Chinese was the dialect around Chang'an (Xi'an); and the current form gained currency after the Yongle Emperor fled his massacres in Nanjing. The southern dialects didn't resist standardization: they standardized earlier and conserve many aspects of old Luoyangese and Xi'anese such as the distinction between /v/ and /w/ and some of the older vowels.

Now, all that said, this is still a good question:

I can understand why different ethnic groups have different dialects, but why is the[re] a difference in the Han Chinese themselves? Presumably the coastal and river cities would be heavily involved in commerce with each other, necessitating a common language? Yet Guangzhou and Beijing speak differently. Why?

As noted by Neubau, the river cities heavily involved in trade with one another do have the same dialect families. In fact, those watersheds and river networks are precisely what define the limits of present Chinese dialects, carried over from premodern limits on transportation. The main mistake causing your confusion is thinking the Chinese traded by sea. They mostly didn't.

Historically, despite their regional differences in genetics, climate, diet, religion, and daily language, the Han were united by some cultural practices, mostly surrounding 孝 which is badly translated into English as 'filial piety'. Sons were expected to care for their parents and to make (at minimum) annual displays of respect to their ancestors; distant voyages had to receive approval (always) from the parents and (if international) from the government. The idea of losing someone at sea (where their body could never be buried with its family or properly venerated) was horrifying, and the idea of its citizens roaming freely on the waves was associated by most governments (usually correctly) with piracy. Permission was often denied. Zheng He's voyages were highly abnormal, highly political (the Yongle Emperor was hunting pretenders to the throne he had usurped), and made possible by Zheng being a Muslim and a eunuch. As soon as the emperor changed, court officials ditched the fleet and dismantled the shipyards.

For the most part, China looked down its nose at traders as parasites, aimed for national self-sufficiency, and used other countries' trade delegations (which they understood as payments of tribute to the Son of Heaven) as a chance to score domestic political points, show off, and make friends, not as a chance to improve revenue and living standards through specialization. There absolutely was some international Chinese trade, as between Ningbo and Japan or between Quanzhou and the Spanish Philippines. There was some domestic coasting trade, as between Ningbo and Shanghai or Guangzhou and Xiamen. But by and large China's trade happened on its rivers and canals. People didn't sail from Guangzhou to Tianjin and need to talk to anyone. People from Guangzhou traded with one another and surrounding towns. Tax shipments were passed north along the Lingqu Canal to the middle Yangtze, where others would pass it to Nanjing, where others would port it up the Grand Canal to Luoyang or Beijing. China's canal network was much preferred for transport as safer and easily controlled; the canals also provided irrigation and flood protection.

I'm not an expert in either Chinese language or culture, but I know that there is discussion about whether Chinese has so many dialects or if those are different languages that share a common writing system.

Prior to the spread of telecommunications (e.g., radio, television) and automobiles, there was no way to transmit audio on such a broad scale. I expect that the central government encouraged the use of a common writing system, but each area read it their own way. For the most part, the government didn't have to worry about how the words were said so long as they read the same way.

Dialects form with small groups of people that are mostly separated from each other. Even with a short 150 years, America developed a large number of dialects of English. England and Japan had a much longer period of time to do the same thing.

With the spread of telecommunications came the implied need for the language to sound the same or, at least, for people to be able to understand a "standard" accent. The greater range of motion afforded by an automobile also helped dialects merge with each other. "Going to the big city" is now a day trip, not necessarily the move of a lifetime.

I expect that the common writing system, regardless of pronunciation, explains why you will see some Chinese programming with Chinese subtitles. Even today, the media provides a "crutch" for Chinese-speakers who don't understand Mandarin Chinese.