How did the helmsman know what he was sailing towards with a big mast in his face?

could the helmsman just lean a bit to the right and see what blinded him

The large steering wheel with handles allows the helmsman to stand off to the side and still have a comfortable grip on the wheel:

From here, if they are standing on the windward side, they should have an unobstructed view straight forward on most small or moderate-sized ships.

The helmsman didn't have to know where the ship was going. He was a sailor who followed the orders of the officer of the deck.

Of course that raises the question how that officer would see, since he was usually also near the stern. For that, see Pieter's answer.

When "on the open sea" the only approaching danger is the weather, which approaches from the direction of the wind. However though modern racing dinghies (and larger vessels) can sail to within about 45 degrees of the wind, earlier vessels were more likely limited to about 55 or even 60 degrees of the wind at close-hauled. When broad-reaching or running the weather is approaching from the vessels rear. The helmsman has a clear view of approaching weather at all of these points of sail.

For seeing out to a further horizon, a crewman posted to the crowsnest on the main mast was standard procedure except in the most vile weather conditions.

When in close quarters, such as maneuvering in a restricted channel, a crewman would have been posted to the bow to perform depth sounding at a very low speed. Frequently also a local pilot would have been brought on board to assist with safe navigation, when such was available.

Finally, note that:

1. The helmsman is located well back of the mast itself, so it obscures little of his vision.

2. The big sail is in the helmsman's face only when the vessel is running downwind. at all other points of sail the sail themselves are well to leeward of the vessel's centre-line.

UPDATE:

It's a misapprehension that the helmsman is responsible for avoiding obstacles. Other crew have the responsibility for that, whether a seaman in the crows-nest, a sounder at the bow in shallow water, or a spotter on the gunwale in confined quarters. The helmsman's responsibility is to play the tiller in harmony with the wind and the waves so as to keep the sails full - while averaging the course as set by the vessel's Captain or Master, and accepting direction from the other assigned crew. This requires great skill and practice, to read the wind and waves so as to not let any sail of the two dozen or more that might be deployed, for example when running downwind. Despite sailing dinghies competitively for many years, and becoming an accomplished dinghy crew, I could never master the tiller-feel required of a helmsman.

The person at the wheel or tiller did not decide how to steer the ship. The person holding the steering device was usually of low rank and would steer the way the officer in charge at the moment told him to. That meant that he would usually hold the control so the ship was going straight at a specific compass heading, until instructed to change course.

If there was someone watching in the crow's nest on the mast or at the bow they could yell to inform the officer of anything important. Other crewmen who noticed something could also inform the officer if it was important. The officer wasn't stuck to the wheel and could move from right to left, front to back, or even climb partway up a mast, to get a better view if he wanted.

Before steering wheels were invented about 1700 and gradually came into use, a whipstaff was used on European ships to turn the tiller and the rudder. And the whipstaff was in a room in the stern castle. The view of the sea ahead was blocked not only by the masts and sails but by the forecastle at the front of the ship and the front wall of the room with the whipstaff.

The steering man was steering blind as far as a view of the sea ahead was concerned. The job of the steering man was to hold the ship on a steady course in relation to the compass needle. Winds and waves and currents were constantly changing and thus turning the ship and he had to fight against them to keep the ship on a steady course. There were also windows high in the wall of the whipstaff room to view the sails and how changes in the wind were affecting them.

When sailors were up on the masts working on the sails the steering man had to be careful not to do something that might shake them off to fall to their deaths.

Where's the Ship's Wheel?

The Manila-Acapulco Galleons : the Treasure Ships of the Pacific

Whipstaff for new galleon

Added 11-09-2020.

I found a scene in a movie which implies that the person at the tiller can't see what is going on or make decisions. Thus some movie makers avoided the anachronism of showing steering wheels on deck before they were invented.

It is a clip of a scene from The Spanish Main (1945), which is set roughly about 1680-1720, as far as this pirate fantasy can be fitted into any date in real history. The captain of a Spanish galleon on the poop deck orders "larboard helm" and an officer beside him takes a speaking trumpet and yells "Larboard helm!" to the lower decks off screen.

And of course the ship is too modern to be called a galleon, but it is impressive enough for popular culture that a steering wheel is not shown on deck.

End of 11-09-2020 addition.

When steering wheels were invented around AD 1700 and came into use the steering person was now up on the deck of the stern castle and had a better view, though partially blocked by the masts and the sails. But the difference in the view didn't matter, because the helmsman at the wheel still steered as directed by the officer in charge, who could move to get a better view and get warnings from other crew members if needed.

On November 20, 1820 First Mate Owen Chase (1797-1869) was the senior officer aboard the whaling ship Essex at the moment when a whale swam toward the ship.

His appearance and attitude gave us at first no alarm; but when I stood watching his movements, and observing him but a ship's length off, coming down for us with great celerity, I involuntarily ordered the boy at the helm to put it hard up, intending to sheer off and avoid him.

The whale moved off after striking the ship the first time, and then made a second charge:

I bawled out to the helmsman "Hard up!", but she had not fallen off more than a point before we took the second shock.

Stove by a Whale

Fifteen-year-old cabin boy Thomas Nickerson (1805-1883) was at the wheel at the time.

In modern ships the bridge is now high up with a good view of the sea ahead, but that doesn't make that much difference to a steering person's duties.

For example, the bridge of R.M.S. Titanic had a good view of the sea ahead. But Quartermaster Robert Hichens (1882-1940), at the wheel on the night of April 14 to 15, 1912, didn't see the iceberg ahead and immediately turn the wheel to try to avoid the iceberg.

Lookouts Frederick Fleet (1887-1965) and Reginald Lee (1870-1913) up in the crow's nest spotted the iceberg ahead.

Nine minutes later, at 23:39, Fleet spotted an iceberg in Titanic's path. He rang the lookout bell three times and telephoned the bridge to inform Sixth Officer James Moody. Fleet asked "Is there anyone there?" Moody replied, "Yes, what do you see?" Fleet replied: "Iceberg, right ahead!"[34] After thanking Fleet, Moody relayed the message to Murdoch, who ordered Quartermaster Robert Hichens to change the ship's course.[35] Murdoch is generally believed to have given the order "Hard astarboard" which would result in the ship's tiller being moved all the way to starboard (the right side of the ship) in an attempt to turn the ship to port (left).[30] He also rang "Full Astern" on the ship's telegraphs.[21]

Sinking of the RMS Titanic

Thus Hichens turned the wheel the way he was instructed by Murdoch.

I once saw a US Navy recruiting ad on TV with a teenage sailor at the steering wheel of an atomic submarine. Atomic submarines can stay underwater for months at a time and often do, and don't have any windows. So neither the helmsman nor anyone else can see anything ahead. The navigational officer instructs the helmsman to steer at the proper heading, and the sonar operators warn if there are obstacles ahead that need to be avoided.

Thus in most European designed ships during the last 500 years or so there has not be much need for the person steering to have a good view of the sea ahead of the ship.

See also this question and its answers: https://history.stackexchange.com/questions/50853/why-did-titanic-need-two-steering-wheels/50864#50864[2]