How did people realize that America was a different continent?

Columbus and everybody else at the time believed that he had discovered the Eastern shores of Asia. Vespucci was one of those amongst the everybody else. An essay by Jonathan Cohen titled, The Naming of America: Fragments we've shored against ourselves, covers the naming of America in some detail. In it, Cohen notes:

The voyage completed by Vespucci between May 1499 and June 1500 as navigator of an expedition of four ships sent from Spain under the command of Alonso de Hojeda is certainly authentic. This is the second expedition of the traditional series. Since Vespucci took part as navigator, he certainly cannot have been inexperienced; however, it seems unlikely that he had made a previous voyage, though this matter remains unresolved. In the voyage of 1499–1500, Vespucci would seem to have left Hojeda after reaching the coast of what is now Guyana (Carew's homeland). Turning south, he is believed to have discovered the mouth of the Amazon River and explored the coast of present-day Brazil. On the way back, he reached Trinidad, sighting en route the mouth of the Orinoco River, and then made for Haiti. Vespucci thought he had sailed along the coast of the extreme easterly peninsula of Asia, where Ptolemy, the 2nd-century Greek geographer, believed the market of Cattigara to be; so he looked for the tip of this peninsula, calling it Cape Cattigara. He supposed that the ships, once past this point, emerged into the seas of southern Asia. As soon as he was back in Spain, he equipped a fresh expedition with the aim of reaching Asia. But the Spanish government did not welcome his proposals, and at the end of 1500 Vespucci went into the service of Portugal.

Under Portuguese auspices he completed a second expedition, which set sail from Lisbon on May 31, 1501. After a halt at the Cape Verde Islands, the expedition traveled southwestward, reached the coast of Brazil, and certainly sailed as far south as the Río de la Plata, which Vespucci was the first European to discover. In all likelihood the ships took a quick run still farther south, along the coast of Patagonia to the Golfo de San Juli n or beyond. His ships returned by an unknown route, anchoring at Lisbon on July 12, 1502. This voyage is of fundamental importance in the history of geography in that Vespucci himself became convinced that the lands he had explored were not part of Asia but a New World. Unlike Columbus, who, to his death, clung to the idea that he had found the shores of Asia, Vespucci defined what had indeed been found — and for this he has been rightfully honored.

Much of this information has been gleaned from Vespucci's letters to his sponsors (also published as Mundus Novus, i.e., New World) where he provides the how he knew that it was not Asia and was instead a new continent that he had discovered:

For the opinion of the ancients was, that the greater part of the world beyond the equinoctial line to the south was not land, but only sea, which they have called the Atlantic; and if they have affirmed that any continent is there, they have given many reasons for denying that it is inhabited. But this their opinion is false, and entirely opposed to the truth. My last voyage has proved it, for I have found a continent in that southern part; more populous and more full of animals than our Europe, or Asia, or Africa, and even more temperate and pleasant than any other region known to us, as will be explained further on.

It was on the 7th of August 1501, that we reached those countries, thanking our Lord God with solemn prayers, and celebrating a choral Mass. We knew that land to be a continent, and not an island, from its long beaches extending without trending round, the infinite number of inhabitants, the numerous tribes and peoples, the numerous kinds of wild animals unknown in our country, and many others never seen before by us, touching which it would take long to make reference.

In other words, Vespucci concluded that he had discovered a continent based on the following facts:

• huge land mass
• extending further south than Asia
• different flora and fauna
• different peoples

A resulting map from 1507 which is often credited as the source for the name America looked like this: