Did medieval people wear collars with a castellated hem?

The basic form of this garment is like the gugel, a hood that protects the head and also covers the shoulders. The precursors for these are Roman paenula or Alpine Kotze made from various types of wool.

It is not exclusive to medieval times, although this basic style became quite fashionable for a while during the later middle ages, first in the lower classes, then in the higher up strata of society.

A recent offering for this type of head and shoulder protection for hunters on sale now:

One of the oldest finds of these is the 'Orkney hood', an iron age clothing with already this type of a seemingly 'ornamental' hem.

Since it is of functionally the same type of a poncho, its advantages are keeping the shoulders protected and warm while allowing for good movement of the arms.

Further, the cloth used is quite water repellent and thus protecting against rain.

This is one minor aspect for the "castellation": when it rains this fringe channels a bit of the water and also allows for faster drying. The effect is not really big and certainly even more diminished for the squarish style depicted in the question.

are often trimmed with a fringe – originally a functional detail, to allow the garment to shed rain, and to dry faster when wet because the fringe acted as a series of wicks to disperse the water – or quills

(WP: Buckskins)

In this clichéd version the effect cannot be that great. If worn in that style in reality, this is an ornamental remnant with a functional past. Here, it is more likely a stereotypical convention for the drawings. Notice that in the first picture only one of three wears this hem. In other words: in this picture 'only Robin wears a hood', and here with some indentations, or dagging or in German lands the often quite complicated zaddeln.
That such a hem might have been used in real life occasionally isn't precluded by the prior statement. Fashion isn't always really concerned with practicality.

Today the most close resemblance for this garment, often without the hood, is often called pélerine (compare the pictures on different language Wikipedia pages), and in liturgical items one sees the resemblance to a chasuble and mozzetta. The German term Gugel in Normannic would be Cagoule, although the modern item under this name looks quite a bit different now and covers the arms with sleeves.

In terms of archaeological finds, the Swedish Bocksten Man, dated 1350–70, presents both possible variants at the same time: a gugel (hood) and the hoodless cloak, in that case worn as a quite semi-circular long mantle, albeit both with a simple undecorated hem, no fringe.

The medieval 'book on the hunt' (Livre de la chasse) is an illuminated manuscript from between 1387 and 1389. In some versions of it we find all these types pictured: with hood and without, with decorated hem or without.

Further development and names:

In the very late medieval and early renaissance the gugel/hood went mainly into three distinct developments. The face covering was now often worn on top of the head, making the rest for a floppy brimmed hat, ending in the 'chaperon'. The head covering itself was separated from the shoulder coverings and receded into a kind of Phrygian cap. The lower part of the gugel was since the 14th century known as the goller (Latin: collare, ) or partlet.

https://en.wikipedia.org/wiki/Paenula
https://de.wikipedia.org/wiki/Kotze_(Kleidung)
https://en.wikipedia.org/wiki/Orkney_Hood
https://fr.wikipedia.org/wiki/P%C3%A8lerine
https://www.museumhalland.se/bockstensmannen/
https://en.wikipedia.org/wiki/Bocksten_Man
https://en.wikipedia.org/wiki/Livre_de_chasse
https://en.wikipedia.org/wiki/Partlet
Chaperons and Liripipes

— Herbert Norris: "Medieval Costume and Fashion", 1999. Shows a relative timeline for dominant styles of the hood.
— Sarah Thursfield: "The Medieval Tailor's Assistant. Making Common Garments 1200 -1500", 2001. Shows the most basic cuts and construction of these hoods.
— Mary G Houston: "Medieval Costume and Fashion in England and France. The 13th, 14th and 15th Centuries", 1939. (Black & white drawing from Livre de Chasse, p170.)