How did passengers keep warm on sail ships?

Fear of shipboard fire is reflected in the tradition of "the smoking lamp," an expression you probably have heard in films even if you've never been on a ship of any sort. On the old wooden sailing ships, the smoking lamp typically was the only source of open flame (outside of the galley) permitted on the boat. Sailors were only allowed to smoke when the smoking lamp was lit (hence its name) and their smoking materials could only be lit off the lamp. The lamp would only be lit briefly and areas of the ship where sailors were allowed to carry their lit pipes or cigars was limited to prevent them falling asleep in their berth while still smoking. They no longer have a physical smoking lamp but the expression "the smoking lamp is out" is still used when safe operation dictates that all smoking materials be extinguished, and that period is ended with the announcement of "the smoking lamp is lit."

Regarding using the dunny on shipboard, in the days of the square-riggers, before modern sail geometry allowed sailing much closer to the wind, the wind always came over the stern (roughly), which meant the bow was as close to downwind as you could get. Some ships had a seat like that in an outhouse suspended over the railing at the bow, which was where you went to answer the call of nature. 'Head' was another term for the bow of the ship, which is how the expression "going to the head" comes to have that connotation.

And I wouldn't be so dismissive of the warmth of the clothes available in the olden days. One reason Roald Amundsen succeeded where Robert Scott failed was that his expedition dressed in animal furs, a centuries-old technology borrowed from the Inuit, while Scott's team wore 'modern' garments of wool and rubberized fabric.

The Atlantic ocean stretches from pole to pole, more or less, and there's generally no good reason to stay in the coldest parts of it.

Consider the 16th-century trade routes:

And also the "clipper route":

You'll notice they didn't spend a lot of time in the high northern latitudes.

Trans-Atlantic passenger travel was not very popular until the advent of the steamer, and yet men and women crossed the ocean periodically, including the affluent.

Trans-Atlantic passenger travel didn't exist before the advent of steam power. It became possible because of steam power.

Before, people had to have very good reasons for traveling. Migrating to the Americas to start a new life, for example. Or for really important business reasons that could not be dealt with otherwise. If you wanted adventure and visit exotic countries, you'd join the navy. Sightseeing was not a good reason.

From what I've been able to gather, a fireplace, or even a stove, was unthinkable outside the galley.

Correct. Ships were highly flammable. It's a collection of dry wood, tar, linen and hemp. You want to minimize fire hazards as much as possible. Many ships didn't even have a galley. There was a box in which fire was allowed for cooking only. Everyone or every group took turns preparing something hot to eat. Passengers sometimes had to pay for that privilege.

Sailors may have been used to it, but what about folks who paid good money for the trip and expected it to be somewhat comfortable? Did they just use a lot of blankets in their cabins?

Passengers didn't have cabins, as there were no passenger ships. They slept where it was possible. On top or in between cargo, for example. Only very rich and/or important passengers would be able to secure a bunk or even a cabin of one of the officers.

As to 'a cabin of the officers', don't expect anything we think of as cabins. The captain had a very large cabin, but he was the only one. The other officers had cabins we would recognize as large cabinets. I visited a museum ship (steam powered, so much more comfortable than most sailing ships) some time ago. The cabin of the captain was huge. But his officers slept in what I thought were cabinets.

People were used to much harsher conditions than we can imagine. Don't forget the climate of that period, it's called the Little Ice age. Today we consider about 21 C room temperature, but that's fairly recent. My mum told me 16 C was room temperature when she was a child, with one heater for the whole house. Which was turned off during the night.

If you felt cold, you had to dress up. Look at pictures of the period: people wear clothes in (a relatively warm) house we would wear outside in chilly weather. Translate that into: it was cold enough inside (heated) houses to dress up for it.

On board ships, it was real hardship. You couldn't turn on the heating - there wasn't any. If you happen to make the crossing in bad weather, you were wet, miserable and cold.

Another item of misery: going to the loo on board. There wasn't one. You had to do your business in a bucket or over the side. Downwind, of course. Hopefully the weather wasn't too bad. If it was: though luck!

It's not a coincidence many people made their last will before departure.

There were several ways to stay warm. Not that any of them were exactly great. First winter travel was rare.

Next is the fact that passengers (not crew) would not really go above deck much. They mostly just traveled below deck. If we're sticking with just passengers, and not talking about crew, and if we're talking about "the age of sail" then mostly the inside of the boat would be quite warm. Body heat and the timber its self would be quite good at keeping people warm. Again if we're talking about passengers here.

Drafty areas (again for passengers) could be covered with a thick blanket to reduce air flow etc. Keep in mind that for passengers, a tip across back then would have been like a 6-week stay in a small closed up prison cell. They wouldn't want to come out much, and interacting with the crew was generally not a good idea.

As for the crew. Well, it would have sucked. All the benefit of being closed up in a small box would be gone. They stayed warm below deck basically by just having lots of people crammed in a small space. Body heat is no joke. Of course the second they went out on deck it would be very cold.

Long ago, in 16 century they used open fire in fair weather (with all possible precautions) on the deck to cook (ref. Morison, Admiral of the Ocean Sea). When the sea was rough, only cold food could be served. Later they used galleys of higher and higher sophistication, but still mostly in the good weather. There was no other way to heat oneself, except with a lot of cloth, blankets etc. Yes the travel was not very comfortable, even for the rich.

They rarely traveled in winter by the way, and for more important reasons than mere cold.

Recently I saw a film on YouTube of some re-enactment of a trip of Vikings in a replica of a Viking ship. The re-enactment was in summer, and they used modern high-tech cloth. But the ship and its equipment was authentic as much as possible. Several people had to drop from the enterprise, and they were picked by an accompanying modern boat and evacuated. Because they could not tolerate the cold. (And being wet for 24 hours day after day.)

Such stories are abundant. One 20th century re-enactment of the Columbus voyage had to be abandoned completely because the crew could not endure the conditions and rebelled. I recall that Columbus own crew was also on the verge of rebellion when America was finally discovered.

My general impression is that people were tougher, and could endure more hardship then most of modern people. (And more people died from various reasons, in particular during the sea travels.)

Remark: I sailed in the North Atlantic myself, in a 40 ft sail boat, in summer. There was no heating of any kind. It was difficult to stand a 4 hour night watch, in ordinary cloths without a special modern high-tech sailing suit. But it was OK to sleep in a sleeping bag in a cabin. Standing these night watches I tried to imagine how people could stay on the deck at night under icy spray, wearing only wool, linen, leather etc. in the old times, and it was hard to imagine.