What did medieval European knights do if they were defeated in battle?

Well, defeated knights would either have been killed in action, or captured, or fled, or retreated in order. Those killed in action wouldn't have to worry about what to do, I suppose - though their suriving heirs might.

Talking of the ordinary knight surviving a lost battle, one who fled or orderly retreated from the battlefield, the next step would depend of the status of their liege lord. If the defeat wasn't decisive, and their side was up for continuing the fight, then they would likely join the regrouping army, and fight again, unless they were wounded or otherwise unpaired for battle. Think the French survivors of Agincourt, for instance.

If their side lost not only the battle, but the war, it would much depend on the good will of the vanquisher. One possibility was to swear fealty to the new sovereign, keep the corresponding feud (perhaps diminished, or with increased obligations?) and be part of the new order. Some, I suppose, would take that but still conspire against the new sovereign; it would depend on their loyalty to the defeated overlord, and on the perceived viability of a restoration of the old order. Plus, an ordinary knight would be under a complex chain of loyalties, and there could be conflict between higher and lower loyalties - perhaps the King wanted to go on with resistance, but the Duke was willing to bend the knee for a new sovereign, or the Viscount was eager to replace the former Duke in the new order. A knight would have to decide whether to remain loyal to his immediate superior, or to the higher overlord, or any intermediate links of the now broken chain...

Here is an account of the fate of the old Anglo-Saxon aristocracy under William the Conqueror:

The class of English aristocrats began to disappear after the Battle of Hastings and the process continued after uprisings against William followed his invasion. Many of the thegn class left England for Scotland and Scandinavia; others joined the Varangian Guard at Constantinople. Those who continued to live in England survived in poverty and reduced circumstances and in an uncertain position, depending on the terms they were able to negotiate with their new lords. The old English aristocrats were relegated to a "kind of appendix." They took a place with the Norman servants of the king, or "among people of depressed condition."

In a more general way, suicide - and especially ritual suicide - wasn't really an option; the religion of the times strictly forbade it, so the practice was discouraged, and there was no established ritual, no seppuku rules. The closest thing to that a knight could and would do was somekind of desperate charge against the enemy (the battle being evidently lost but not tottaly over, to search death by exposing oneself to the greatest danger, attacking in small numbers against an overwhelming enemy instead of fleeing): a "suicide" attack, which would certainly not be considered suicide by the mores of the times.

Captured knights, as JMS answer points out, were at the mercy of their captors. The only choice they had was to cooperate with the captor if there was such option (writing or speaking to relatives or underlings, telling them to comply with the captor's demands for ransom or political compromise) or to refuse to cooperate. Cooperation was not considered dishonorable of itself; neither was complying with political demands and then, when freed, to recant the promised concessions.

Details vary from century to century and area to area, but ransom was a common practice, cf Richard I Lionheart.

It was considered generous to waive the ransom, or to consider that horse, arms and armor were enough, but taking ransom was not considered dishonorable. After all, it encouraged the healthy return of noble captives.

Question: What did medieval Knights do if they were defeated in battle?

They did whatever their captors told them to do. Medieval captives were entirely at the mercy of their captors. Some were held hostage for prisoner exchanges, some ransomed, some were kept indefinitely, some coerced to change objectionable opinions or sides, and of course some were just murdered. Over the centuries pretty much anything and everything was done. However, the capturing party could gain some kind of advantage, they did.

Exchanged

• Robert, Earl of Gloucester was exchanged for King Stephen captured at the Fist Battle of Lincoln 1141.

Ransomed

It was lucrative for Kings and important knights to be ransomed, sometimes for huge sums so this was a pretty big incentive.

• Richard the Lion Heart - 100,000 pounds of silver, 2 or 3 times the annual income of the crown of England at that time.
• Bertrand du Guesclin - 100,000 francs.

Imprisoned

• Charles of Orleans, after the battle of Agincourt in 1415 was held by the English for 25 years. He was deemed too dangerous an enemy by King Henry V to be ransomed to France.

Coerced

• The capture of Charles de Blois led to the Treaty of Westminster, (1356) then he was ransomed, and restarted the war.

Edward III signed the Treaty of Westminster on 1 March 1353, accepting Charles of Blois as Duke of Brittany if the latter undertook to pay a ransom of 300,000 crowns, and that Brittany signed a treaty of alliance "in perpetuity" with England, this alliance to be sealed by the marriage of the Montfortist claimant John of Montfort (son of the earlier John of Montfort) with Edward's daughter Mary.

• the capture of John II of France lead to the Treaty of Brétigny (1360) and then he was ransomed.

Treaty of Brétigny, (1360) Treaty between England and France that ended the first phase of the Hundred Years’ War. Marking a serious setback for the French, the treaty was signed after Edward the Black Prince defeated and captured John II of France at the Battle of Poitiers (1356). The French ceded extensive territories in northwestern France to England and agreed to ransom John at a cost of three million gold crowns, while King Edward III renounced his claim to the French throne. The treaty failed to establish a lasting peace, and the war began again in 1369.

• Harold Godwinson of England was captured by William of Normandy in Normandy before the latter invaded England in 1066. As a price for Harold's release William had Harold recant any claim to the English throne. Of course Harold who fought William at the battle of Hastings over that throne didn't find his previous vow to be a binding one.

Murdered

• Elder half-brother of the eventual king of England Edward the Confessor. Earwig, claimant to the English thrown, was betrayed by his one time ally and future in-law(brother Edward's wife's family), Earl Godwin. Earwig was turned over by Earl Godwin to his enemy (the Dane Cnut) who tortured and murdered him, 1017.

• Most of the surviving French knights at the battle of Agincourt in 1415 were put to death by the English. King Henry's French captives outnumbered his own forces, and the English feared they were so large a number they still posed a danger to the army; so the English killed them all.