Why were battles massed until the late 1800s?

In a word, firepower, both volume and accuracy. Refinements to weapons technology came into play by the mid 1800's, while field tactics (always slower to change than weapons tech) remained in the 1700's... at least until the high casualties were totaled up.

The standard soldier's sidearm by 1800, the flintlock smoothbore musket, had a low rate of fire, somewhat low reliability from the flint ignition (especially in damp conditions), and a fairly short effective range, as well as low accuracy. Effective range... 50-75 meters. This accounts for volley fire - one giant shotgun, which also made for a formation similar to the pike square formation that had been popular when pole arms were prevalent.

By the 1860's, the musket had evolved into the rifle - expanding bullets like the Mine Ball speeded up the reload, the percussion cap increased the reliability and also made for a faster reload. The rifles of the US Civil war: Springfield, Enfield, Richmond, etc... had a 200+ meter effective range,and were considerably more accurate than the musket. There was also the first practical sniper rifle, the Whitworth, with it's elongated bullet, that could be effective out to 1000 meters.

A few breechloading rifles with faster reload (that could be done lying down) were coming into use, such as the Sharps, plus the early Colt revolvers that made hit and run raids very effective. A very few Henry repeating rifles with the then new metallic cartridge made it into that war. These had minimal effect on that war, they were more a view into the future.

Artillery increased in both range and effectiveness. As opposed to the standard Napoleon, there was the Parrott Rifle, a rifled cannon with a reinforcing band around the breech, plus the 3" Ordnance Rifle that used stronger steel on the breech, both with a range of around two miles. More accurate, longer range, they could now target massed formations with a reasonable expectation of success.

The US civil war was a confluence of 18th century formations and tactics, with rapidly accelerating military technology, and the result was wholesale slaughter.

This carried forth into WW1, with the machine gun, aircraft, and very long range explosive artillery wreaking havoc on the massed charges that were attempted in trench warfare. The butcher's bill was correspondingly high.

Artillery fire was also far less effective back then. So there was little need to spread out.

Because "communications" (as defined below), didn't encourage spreading out until the late 19th century.

The first form of "communications" was weaponry. Until the late 19th century, "guns" such as muskets and early rifles, were single shot weapons that could be fired only a few times during the battle, in its preliminary stages. Most of the serious fighting was done with blade weapons such as bayonets (or sabres for cavalry), which required mass action.

It was the discovery of "repeating" weapons in the late 19th century that changed that. First the repeating rifle, then "machine" guns, or at least predecessors such as the Gatling gun or Maxim needle gun. If you had guns that could fire, say 20 shots a minute with reasonable accuracy, bayonet charges diminished in importance, and "spreading out" started to make sense.

The second issue of "communications" was the standard one. Radio was developed in the late 19th century, and it was much easier to communicate orders to "spread out" units after, than before, radios. Ditto for telephones, or variations such as walkie-talies after 1876. Air power also played a factor starting with World War I; airplanes could drop flares signaling points of attack, concentration of fire, etc.

There are a number of factors:

As noted in the question, spreading out would make you less likely to be hit BUT it would also make it less likely that you would hit the enemy. Battles would take much longer, would tend to be less decisive and potentially end up as hand-to-hand engagements when both sides ran out of musket & cannon balls.

Even with muskets, many battles hinged on a bayonet charge into the enemy, where the mass of soldiers is as big a part of the 'impact' as the sharp, pointy thing on the end of the gun barrel. So keeping the men together as a unit allowed a concentration of firepower at musket range and a heavier impact in the charge.

Another factor is control. Most of the soldiers in these armies were not professionals. Keeping them going in the right direction (when they really want to be going in the opposite one) is far easier if they are marching together in ranks rather than spread out across the field. It also made identifying units easier for the commanders of the army so that orders could be given to the correct subordinates to get those units to the right place.

In medieval and ancient battles, only archers and slingers could hit the enemy at a distance. Most soldiers fought the enemy with hand weapons, spears, swords, battle axes, etc. etc.

Armies with warriors or soldiers packed more closely together would have more men per unit area and so have the local advantage of numbers and be more likely to win in hand to hand combat. At least until the warriors became crowded together too closely to fight effectively.

Thus ancient and medieval armies tended to have their warriors in each unit packed as tightly as was practical for effective fighting.

Musket carrying soldiers in early modern armies tended to march and maneuver closely together so that the soldiers would be used to doing as their comrades were doing and would be less likely to think independently and run away in terror like any rational person would.

Armies also had to use dense formations to enable the officers and non commissioned officers to command and control their men. They didn't have field telephones or walkie talkies or portable radios or cell phones or the internet or anything except their voices to communicate with their men.

The officers and non commissioned officers had to be within voice range for their men to hear their orders, and in sight for their men to see what they were doing and (it was hoped) be inspired by it.

And ways were invented for leaders to become louder and more visible.

Flags, the banners and standards of feudal armies and the colors, standards, and guidons of modern military units, were carried into battle flying high and visible at greater distances than an individual leader might be.

And military music was used. Drums and fifes, bugles and trumpets, the field music. They were often used to signal the marching pace. They were often used to give signals in battle that might be heard further than an officer's voice.

And so armies from ancient times up until late in the 19th century fought in close formation not so much because it might have sometimes been tactically superior, but because it was the only way to have command and control over military units.

The rule was no close order formation, no cohesive army, until the 20th century.

There are two main reasons for the concentration of massed soldiers:

• The only defence of infantry against cavalry was the ability to form a square, whether open like the French and British or closed as was most common east of the Elbe. Seconds mattered when half a thousand horse and riders were charging at 30kph or faster with malicious intent, so being very close to your comrades was essential to survival.

  Note that, contrary to popular belief, the 1000-man British battalions typically fought in 4 ranks rather than 2 for most of a battle, because forming square from a long 2-rank line just took too long. Only late in a battle, after suffering casualties and with opposing cavalry often winded or hors de combat, would they adopt 2-rank formations to deiver a coup de grace.

• A battalion of 500 men firing 3 rounds per minute each (an average rate) was firing 1500 rounds per minute, comparable to a modern light machine gun. Like such a machine gun, this fire was more leveled than aimed, and achieved its effect from concentration into a small space. This concentration of fire was achieved by concentrating its source - the musketeers firing.

It's important to note that the volume of battalion fire hitting a target varied significantly depending on both the nationality and experience of the unit. Inexperienced conscripts such as Austrian Landwehr and third battalions often hit with less than 5% of balls fired, while Guardsmen of all nationalities probably scored in excess of 20%. Factor in the chance of multiple hits, and survival in a 18th or early 19th century firefight was not unlikely in most cases.

Let's also look at skirmishers - the perfect example of OP's argument. Other than elite (and typically non-Guard) units of all nationalities, units never attempted to break down completely into skirmish line. The reasons revolved around both the need for defence against cavalry (mentioned above), but also because maneuvering 500+ men operating in pairs across several hundred square yards of terrain required exceptional training of not just men, but also non-coms and officers. Only in Davout's III Corps of the Army of Germany was it widespread for this degree of training to exist, and for most nationalities the lesson was not learned until deep in the First World War.