How did the ancestors of proto-Malagasy in South Borneo reach Madagascar?

Question:

What is the current state of our knowledge regarding potential routes and vessels the Asian ancestors of the Malagasy may have used to reach Madagascar, and some key items of evidence supporting any hypotheses that may exist?

Answer: Most likely from Maritime Southeast Asia, via Malay-controlled trade ships on Indian Ocean voyages during 1st millennium CE.

A pioneering study was published in 1951 by Otto Christian Dahl proposing a close genetic link between Malagasy of Madagascar and Ma'anyan of Borneo.

Ethnogenesis

Using linguistics, in particular the "Barito and Dayak languages of South Kalimantan", academicians have identified ancestors of Madagascar as "Indonesian". Let's be careful here.

Borneo is Asia's largest island, so to be more precise, let's narrow it down to Barito River of Borneo. Today, this is in Kalimantan, Indonesia. Malaysia's Sabah and Sarawak is on the northern side of Borneo. The nation-states of Indonesia and Malaysia is a 20th century creation.

I have seen at least one academic identify the ancestors of Proto-Malagasy as "Indonesian". Although the reference was to Sulawesi, Indonesia, it is confusing and misleading. During pre-colonial times, there was no political border in Borneo and the indigenous people of Borneo is an eclectic mix of many tribal groups. It should simply be: speakers of South-east Barito group, of South Borneo. (Even today, the political border of Malaysia-Indonesia in Borneo is still porous).

Migration from South Borneo or Maritime Southeast Asia?

(Image source)

This is the most likely routes, from south Borneo and either going west (via Sunda Strait) or going northwest via Strait of Malacca. The map seems to say the "blue arrow" is the correct one. More importantly, it is directly from south Borneo (Banjarmasin, capital of Kalimantan). But why does it have to be directly from south Borneo? I believe the trade routes (in red) is more probable.

In fact, it does not have to be directly from south Borneo. The migration could have been anywhere from Maritime Southeast Asia (not necessarily directly from south Borneo). The trade routes of Southeast Asia and India, then Madagascar, is more realistic (practical?) than a straight voyage directly to Madagascar. The Malay sailors (more below) would have taken advantage of the Southwest-Northeast monsoon (see map).

On the linguistic front, in addition to Barito language, Malagasy culture and language has loanwords from old Malay, Javanese and Sulawesi and even Sanskrit. Most pertinent:

Malay loanwords in Malagasy outnumber by far loanwords from other Indonesian languages, it is likely that most Sanskrit loanwords in Malagasy have been borrowed via Malay. But for reasons of documentation, this is impossible to demonstrate. Both Old Malay and Old Javanese had a large amount of Sanskrit loanwords.

Source: The Indonesian migrations to Madagascar: Making sense of the multidisciplinary evidence, 2006, p.7.

Recent articles have stated that ships plying the pre-colonial trade routes were mainly operated by Malay traders -- keeping in mind that political borders were different during 1st to 12 century CE. So, in short, the migratory routes of Austronesian language and culture was most probably from any major port on western side of Maritime South East Asia (not just south Borneo).

... another theory proposing that links between southeast Asia and east Africa were primarily established by Malays who were the agents behind the migrations of Southeast Barito speakers to Madagascar. ... This theory supports the fact that neither the Southeast Barito Dayaks nor the Malagasy have ever developed the seafaring skills required for the ambitious voyages that their ancestors made, while the Malays have.

Source: Peregrine, P., Ancient Human Migrations: A Multidisciplinary Approach (2009), p. 161

Canoes vs Ocean-capable Vessels

I am not aware of evidence that any outrigger canoes that may have been available to the settlers of Madagascar would have had performance characteristics similar to those of (more modern?) Polynesian canoes, or that their navigational skills matched the highly-developed skills of Polynesians seafarers (as evidenced by the Marshall Island stick charts, for example).

These ships were not "outrigger canoes", they really were capable of long-distance voyages. Image below "A ship under full sail, panel no. 86, Borobudur":

It is understandable why the romantic image of Malay or Austronesian canoeist from ISEA has persisted (emphasis mine):

Once lip service was paid to the remarkable maritime adaptations and sailing talents of Southeast Asian people, the issue attracted little additional attention. Many pages were devoted to the distribution of such features as outriggers (single or double) and sail types from Southeast Asia into Oceania and, via India, to Eastern Africa and Madagascar, but cultural diffusionists were content to discuss the origins of such features without consideration for the historical context of Southeast Asia. The unsuspecting scholar was therefore largely left with the romantic but unsubstantiated idea that fearless Austronesian-speaking people had paddled halfway around the planet on flimsy boats. Given this obsessive focus on origins and diffusion, with no consideration to the full historical processes, it was difficult to figure out why and how the Austronesian speakers, who had first spread into insular Southeast Asia and the Pacific, subsequently leaped west across the Indian Ocean, reached East Africa, and peopled Madagascar.

Only recently has the evidence emerged, from past few decades:

For a long time, historians restricted their role to examination of regional peddling trade carried out in small ships. Research on ships and trade networks in the past few decades, however, has returned considerable agency to local societies, particularly to Austronesian speakers of insular Southeast Asia, from proto-historic to early modern times. As far in the past as two thousand years ago, following locally developed shipbuilding technologies and navigational practices, they built large and sophisticated ships that plied South China Sea and Indian Ocean routes, as documented by 1st-millennium Chinese and later Portuguese sources and now confirmed by nautical archaeology. Textual sources also confirm that local shipmasters played a prominent part in locally and internationally run trade networks, which firmly places their operations into the mainstream of Asian global maritime history.

Source (image & text): Ships and Shipping in Southeast Asia, 2017; Oxford Research Encyclopedias (online). See also other sections available in ORE on Navigation Techniques and Shipmasters in Southeast Asia: Seafarers and Entrepreneurs.

OP had another question (in comment):

What are "proto-Malagasy"?

From Wikipedia on History of Madagascar (emphasis mine):

The Southeast Asian origin of the first Malagasy people explains certain features common among the Malagasy, for instance, the epicanthic fold common among all Malagasy whether coastal or highlands, whether pale, dark or copper skinned. This original population (vahoaka ntaolo) can be called the "Proto-Malagasy" . They are the source of:

• the Malagasy language, common to the whole island, which shares many common roots with the Barito and Dayak languages of South Kalimantan such as Ma'anyan,23 and belongs to the Austronesian language family, with closest affinity to the Bornean languages.

• Malagasy cultural traditions shared with Austronesians of Taiwan, the Pacific Islands, Indonesia, New Zealand, and the Philippines including ancient customs, such as burying the dead within a canoe in the sea or in a lake, the cultivation of traditional Austronesian crops such as taro or saonjo, banana, coconut, and sugar cane, traditional architecture with a square house plan, music and musical instruments such as the Antsiva conch, the hazolahy drum, the atranatrana xylophone, sodina flute, or the valiha tube zither,[24] and dance, including the "bird dance" found both in central and southern regions.

Finally, on Indian Ocean trade voyages during Southeast Asian maritime history, in addition to book and article cited above, this is also useful on shipbuilding and the "technical" side: The Southeast Asian Ship: An Historical Approach, 1980. To end, I don't have a written source but it is generally accepted (my own opinion) the Indian Ocean voyages, if well planned, can be conducted in a week (each direction).

Incidentally, I answered a previous question on diaspora of Austronesians from southeast China region, and Taiwan, into Island Southeast Asia.