Why did people stop building square-rigged ships?

Let's first recap points of sail.

On a dead run (and for most rigs also a broad reach) it is only possible to travel at slightly less than the wind speed. This is because the vessel is being powered only by wind drag, with no lift component. However in a light breeze even that can be difficult, so it is imperative to catch of much of that light breeze as possible - especially at heights where reduced surface friction makes the breeze slightly greater.

Square rigs are the best at catching light breeze at maximum height above the water. (A square sail at the top of a mast, hanging from a yardarm, captures 4 or 5 times as much wind as the triangular head of a fore-and-aft sail of the same height.) For tasks such as long-haul cargo transport over distances at which one can mostly sail on a run or reach, the square rigger is perfect. Trade Winds or Westerlies are used depending on one's direction of travel.

However square rigs are only capable of pointing to within about 60 degrees of the wind, and that only in light to moderate breeze

Next, note that as a vessel points higher into the wind there is more and more wind force pushing the boat over into a heel. In a dinghy this can be countered by moving the crew weight to the gunnels or even hiked out, as here:

A heavier, deeper, keel also assists in countering the heel force (in addition to improving the pointing). However a sailing vessel with a deep keel cannot use a shallow harbour; and keeping the crew on the rails is impractical on long expeditions, for obvious reasons.

When competitive day-racing began around the turn of the 20th century, the triangular race-courses favoured made upwind-pointing (and, for smaller boats, the ability to plane) critical design decisions. This resulted in the preponderance of fore-and-aft rigs, such as lateen and sloop, on these boats.

Returning to the question - production of square-rigged ships ceased when their primary use - long-haul cargo transport - was superceded by coal and then diesel-powered vessels. Their poor upwind handling was the primary reason why that sailplan was never popular for day-racing. It is unlikely that crew requirement was critical in that regard, when one looks at the huge ships that competed for the America's Cup races in the late 19th and early 20th centuries:

• Defender 1895 (crew at east 35, counting from image):

• Reliance 1903 (crew 64):

A final note on the reason for favouring triangular race-courses in day racing: Because the upwind boat is always favoured, due to the ability to cover the opposing boat, interfering with its wind and slowing it down. This advantage then alternates between the leader(s) headed upwind and the trailer(s) headed downwind.

A classic example of this was the 1983 America's Cup best-of-seven final. Despite having a significantly slower hull, Dennis Connor managed to win three races and force a seventh through superior helmsmanship. The final upwind leg, with fifty real and faked tacks, is regarded by many as the epitome of fine sailing.

On these race-courses, a fore-and-aft rigged ship compensates for the absence of square-rigged sails by employing a spinnaker: A (usually brightly coloured) over-sized balloon of a sail designed to capture as much wind as possible, as high up the mast as possible. It employs a spinnaker-pole as a temporary yardarm to achieve these aims. In that sense, the square-rig is not truly dead - just modified.

Update:

Rebuttal of the claim made below by another author. From the link above, here repeated, to the Royal Yachting Association's description of Points of Sail:

Beam Reach – This is the fastest and easiest point of sail. The wind is on the side of your boat (beam) and you’ll sail with your sails out half way.

Run – With the wind directly behind you this is the trickiest point of sail to steer as it can be quite unstable. On a run your sails can be let out on opposite side of the boat to catch the wind (sailing goosewinged) or a big sail called a spinnaker can be set.

Note the point being made that beam reach is the fastest and easiest point of sail. Also that a run ... is the trickiest point of sail to steer as it can be quite unstable. Their words not mine.

The answer below claiming otherwise is incorrect.

The reason is that at point of sail below a beam reach there is very little (broad reach) to zero (dead run) lift force possible from the sails. The boat cannot travel faster than the wind because at these points of sail it is impossible to keep the sail full except when travelling slower than the wind. In an even slightly unsteady breeze one cannot even attempt approaching the actual wind speed because if a lull ever empties the sails, this immediately halts the boat by collapsing and reversing the sails.

At very low windspeeds it is imperative to never lose your momentum, and come to a dead halt. One might have to wait for a gust, perhaps a substantial one, to break the skin friction with the water and get moving again. Sailing at a dead run or broad reach in a light, unstable breeze threatens this constantly. This is why sailors so dreaded the doldrums and horse latitudes in the age of sail: not only was the wind light, one had to take such care sailing to it, for very long stretches of time. They would be spending the entire day scrambling up and down the rigging (gently, to not disturb the sail trim) to trim the sails so as to never collapse on a lull or shift.