Why did the Royal Society in 1771 believe that a continent further south than Australia should exist?

The claim was made by Alexander Dalrymple in his book An Account of the Discoveries Made in The South Pacifick Ocean, Previous to 1764 (An ebook version is available from Google Books).

Dalrymple's belief was based primarily on his translations of Spanish documents captured in the Philippines in 1762, in particular, those describing Luis Vaez de Torres' account of a voyage south of New Guinea. He supplemented the information in those translations with records of Dutch explorations.

These described sightings of land which, when plotted on a chart, convinced him that they must be parts of a massive continent in the southern latitudes.

Dalrymple initially published his research in two volumes. The first was primarily his translations of the captured Spanish records (which includes the chart), and the second contained the Dutch voyages. Copies of these volumes can be viewed and/or downloaded as pdf files from archive.org:

• An Historical collection of the several voyages and discoveries in the South Pacific Ocean - Volume I - Being Chiefly a LITERAL TRANSLATION from the SPANISH WRITERS
• An Historical collection of the several voyages and discoveries in the South Pacific Ocean - Volume II - Containing the DUTCH VOYAGES

EDIT: 27 August 2018

Having read the comments below, I should probably make it clear that Dalrymple's claims were, almost entirely, wrong. Indeed, even by a charitable interpretation, some were overly optimistic at best! Nevertheless, in the absence of any evidence to the contrary, they were pretty much the best 'evidence' available to the Royal Society at the time. They also conformed to, and reinforced, a belief in Terra Australis Incognita that dated back to Aristotle.

There was a long-standing antipathy between Dalrymple & Cook. After his first voyage, it was clear that Cook was rather sceptical about the existence of Terra Australis Incognita, and in particular the claims made by Dalrymple. Nevertheless, he remained open to the possibility that the continent remained to be discovered in Southern Latitudes. In the journal of his first voyage, he wrote:

I think it would be a great pity that this thing which at times has been the object of many ages and nations should not now be wholly cleared up, which might very easily be done in one voyage without either much trouble much trouble or danger or fear for it.

Captain Cook's journal during his first voyage round the world made in H.M. Bark "Endeavour", 1768-71

So even after Cook returned from his first voyage, Dalrymple's claims had not been disproved, and remained a tantalising possibility.

Some of the sightings that Dalrymple believed to be part of the great southern continent (and plotted as such on his chart) turned out to just be islands (such as the Island of South Georgia). The South Sandwich Islands were partially explored by Cook during his second voyage. He still thought they could have been 'a point of the continent' and stated:

"That there may be a continent, or large tract of land, near the Pole, I will not deny; on the contrary, I am of the opinion there is; and it is probable that we have seen a part of it."

going on to conclude:

"The greatest part of this Southern Continent (supposing there is one) must lie within the Polar Circle where the sea is so pestered with ice that the land is thereby inaccessible. The risk one runs in exploring a coast in these unknown and icy seas, is so very great, that I can be bold to say that no man will ever venture [by sea] farther than I have done, and that the lands which may lie to the south will never be explored. Thick fogs, snowstorms, intense cold and every other thing that can render navigation dangerous one has to encounter, and these difficulties are greatly heightened by the inexpressibly horrid aspect of the country, a country doomed by nature never once to feel the warmth of the sun's rays, but to lie for ever buried under everlasting snow and ice.

The natural harbours which may be on this coast are in a manner wholly filled up with frozen snow of a vast thickness, but if any should so far be open to admit a ship ... she runs the risk of being there for ever".

The three voyages of Captain James Cook round the world, Volume 4

So Dalrymple's claims were finally disproved, and Terra Australis Incognita (what we now know as Antarctica), was shown to be much smaller than most people of the day had believed.

But, of course, that is exactly why Cook's voyages were called 'voyages of exploration'.