When the Portuguese began the exploration of the West African Coast, did they believe the Indian Ocean was an enclosed sea?

Definitely not a straight answer but relevant nonetheless: There is (arguably weak or solid, depending on what certain cultural priors you adopt) circumstantial evidence for an historical lighthouse in Mahabalipuram, India that might or might not have been described in some Portuguese maps:

This is explained in a relatively recent video by the amateur indian archaeologist and YouTuber Praveen Mohan

gktscrk answers the question directly but, although it is true that there were legends and fears, and simple minded sailors, it is absolutely false that the elite did not expect to find anything valuable beyond the Bojador, or that only India was a valuable objective.

How significant was the Fall of Constantinople as an event leading to the Age of Exploration?

Read specially the answers from me and Pieter Geerkens.

In short:

• The Portuguese already knew that Gold and Ivory would be found on the coast of west Africa. And that if they got that trade, the Moroccans would lose it. Even IF a way to India could NOT be found, they had enough reasons to go.
• The King had great income as Great-master of the Order of Christ, and he could not just pocket the money without loss of face: thus the navigations had stable administration and financing.

What a wonderful question! Dom Henrique is one of the chief characters of this story, and that means we need to explore him more than the general Portuguese knowledge. These explorations as a whole were a manifestation of the soul of this great man who was well beyond his contemporaries in what he wanted and expected from his sailors. I'm basing this answer on Bradford's 'A Wind from the North'.

In short, the answer is that when Dom Henrique started with the voyages of exploration, he was unsure about what would be found, but by the end of his life he was sure there was a way around Africa, as evidenced on the map of Fra Maura. However, for the average sailor the religious connotations of sailing towards 'the end of the world' and their possible demise there were far more important than what else was there.

It doesn't look as Ptolemy's map was in common circulation though his other works were; Dom Henrique's resolve may also have been strengthened by hearing about the Phoenician circumnavigation of Africa. Yet, the overarching purpose of these missions was to learn more and to discover—often in these cases the final goal may come unexpectedly.

1420's

Firstly, far more important than a map of the world was the understanding that the sailors of the time had. For them, Cape Not was the beginning of the end of the world:

Quern passar o Cabo de Nam, ou tornara ou nam.
He who would pass Cape Not, either will return or not.

Only a touch further south lay Cape Bojador which was the limit of the 'known' world, and the two things beyond it were "the boiling sea and the Ocean of Darkness":

At that point it seemed as if the waters ran downward in a curve, so that, it was said, no ship could ever sail back. The winds drew always from astern, so that even if the curved ocean was a myth, the fact remained that a square-sailed vessel had little chance of beating her way north again.

Bradford speculates that the tale of the Phoenician circumnavigation of Africa was known to Dom Henrique—from the scholars who ended up gathering at Sagres—which would possibly be a better medication than any map.

Knowledge of Ptolemy

What makes a survival of the Ptolemaic map in Portugal or for the Portuguese suspect is that Azurara, the chronicler of Dom Henrique's life, writes:

"he desired to know what lands there were beyond the Canary Isles and a cape called Bojador. For at that time there was no knowledge, either in writing or in the memory of any man, of what might lie beyond this cape."

Yet, Ptolemy is mentioned in specific terms, however (all emphasis and addition original; "he" refers to Dom Henrique):

We know, in fact, that he suspected there were further islands or land masses in the Atlantic from the chronicle of Diogo Gomes, a ship's captain who spent many of his years in Prince Henry's service, and who was with him when he died. "The Prince wished to know," he wrote, "about the western ocean, and whether there were islands or continents beyond those that Ptolemy described. [The Alexandrian astronomer's Guide to Geography was a standard reference book of the period.] So he sent caravels to search for lands."

Common Understanding

The main problem, however, was not whether a way was shown on a map. Dom Henrique's sense of the unknown prompted him to send the ships out; he was expecting to find new things and was willing to wait out until he got results.

What was more difficult was finding good captains and brave sailors to sail that way. At least before the discovery of Maderia, no riches or wealth was expected from the south given it was the traditional Christian 'end of the world'. Indeed, many sailors (perhaps more superstitious than the captains or Dom Henrique and also likely to profit less than the leaders of the expeditions) were very reluctant to proceed to the south:

The Sea of Obscurity, or the Ocean of Darkness, it was called. There dwelt monsters, and the sun stood so high overhead that the water boiled. The men who lived on the outermost edge of the African shore were known to be burned black by the sun, so how could man go farther without being roasted alive? ...

"To sail south," murmured the sailors. "What is the point of that? How can we possibly pass beyond the bounds that are established? What profit can the Prince gain from the loss of a few poor sailors?..."

Bradford notes the dialogue is not made up, but he doesn't specify a source. Azurara is brought out as one example, but Bradford has used other chroniclers as well.

The desolate nature of the desert next to the Atlantic in that part of the North African landmass would not have incentivized the Portuguese any more, at least not until they got past those to the rich delta of the Senegal. Cabo Branco was only reached in 1441, and it's near there that some trade was conducted with local tribesmen, but that was a good two decades into the exploration progress.

Lastly, the specific virtue of the Prince as the overseer of these voyages is brought out with respect to these world-views:

"The Prince always welcomed the captains of his ships with great patience, never disclosing any resentment, listening graciously to the stories of their adventures, and rewarding them as men who were serving him well. But immediately he sent them back again to repeat the voyage — either them or others of his household — impressing upon them, more and more strongly, the mission he required them to accomplish ..."

1460's

Towards the end of his life, the Prince was quite certain that Africa could be rounded. The map by Fra Maura incorporated knowledge that the Prince's voyages had uncovered, and that shows the Cape of Good Hope (described below):

Bradford describes the making of this map:

The map was finished in the spring of 1459, and it is evidence of the information that Henry had collected from all over Africa that the Cape of Good Hope, called the "Cavo di Diab," is shown on this world map. Forty years before Vasco da Gama rounded the cape, either the Prince or his cartographers had heard of an Indian sailing vessel that in 1420 had rounded the cape from the east after a voyage of 2,000 miles.