What laws of war existed before the modern international treaties and conventions?

One modern document that is considered a body of laws concerning warfare and predates the Geneva Conventions, is the "Instructions for the Government of Armies of the United States in the Field, General Order № 100", issued by President Lincoln in April 24, 1863. It was prepared by Francis Lieber and its commonly called the Lieber Code or Lieber Instructions.

It's contents are:

Section I. Martial Law - Military jurisdiction - Military necessity - Retaliation.

Section II. Public and private property of the enemy - Protection of persons, and especially of women, of religion, the arts and sciences - Punishment of crimes against the inhabitants of hostile countries.

Section III. Deserters - Prisoners of war - Hostages - Booty on the battlefield.

Section IV. Partisans - Armed enemies not belonging to the hostile army - Scouts- Armed prowlers - War-rebels.

Section V. Safe-conduct - Spies - War-traitors - Captured messengers - Abuse of the flag of truce.

Section VI. Exchange of prisoners - Flags of truce - Flags of protection

Section VII. The Parole

Section VIII. Armistice - Capitulation

Section IX. Assassination

Section X. Insurrection - Civil War - Rebellion

Here's an example of its articles, article 88 concerning spies:

Art. 88.

A spy is a person who secretly, in disguise or under false pretense, seeks information with the intention of communicating it to the enemy.

The spy is punishable with death by hanging by the neck, whether or not he succeed in obtaining the information or in conveying it to the enemy.

The Lieber code was issued during the American Civil War and is a precursor to the Hague Conventions of 1899 and 1907. The term "Geneva Conventions" usually refers to the Fourth Geneva Convention (1949) and if that's how you used it then it's not only predated by the Lieber Code and the Hague Conventions, but also the three other Geneva Conventions:

1. First Geneva Convention for the Amelioration of the Condition of the Wounded and Sick in Armed Forces in the Field, 1864
2. Second Geneva Convention for the Amelioration of the Condition of Wounded, Sick and Shipwrecked Members of Armed Forces at Sea, 1906
3. Third Geneva Convention relative to the Treatment of Prisoners of War, 1929

Taking a leap back in history, Frédéric Mégret examines the "laws of war" during the Crusades, in A Cautionary Tale from the Crusades? War and Prisoners in Conditions of Normative Incommensurability and concludes:

The conclusion is that an overarching law of war is not necessarily a pre-condition to restraint in warfare, as long as a number of other circumstances exist such as a strong imperative of self-restraint.

The following quote is from Chapter B, The Impact of Traditions of Restraint in Warfare, and gives a (rough) idea of the combatants' approach to warfare:

First, the usages of war were at very different stages of development even where and when they were held to apply. Violence was endemic in much of Europe. The laws of war in Christendom were still in early stages of formation and were restricted both functionally (e.g.: certain weapons such as the arch bow or certain practices such siege warfare or the division of the spoils of war) and personally (i.e.: governing relations within a warrior class rather than standing for some more general concept of humanity). It was widely considered that victors had a right of life and death upon the captured, so that almost any atrocity was conceivable. Rules such as those applicable in times of siege certainly seem quite gruesome in relation to contemporary norms. As far as Byzantine Empire is concerned, it is fair to say that there was hardly any tradition that resembled the laws of war, although the Empire did in due course develop practices relating to prisoners, including the idea of their liberation after the end of a conflict

Islam, by contrast, did possess a more sophisticated tradition of what was to be done in warfare in general, and how to treat prisoners in particular, founded on a strong concept of restraint. The Qur’an explicitly enjoins believers to “fight in the way of Allah with those who fight with you, and do not exceed the limits”. Instructions had been given by Mohammed’s successor Caliph Abu-Bakr (573-634) to the first Syrian expedition that civilians and civilian property should be respected. The 1280 Villiyet was an ambitious attempt at codifying a by then century old tradition of restraint in war, which prohibited torture and mutilation of prisoners as well as perfidious and disloyal actions. It also more explicitly stipulated to “fight with (enemies) until there is no persecution … but if they desist, then there should be no hostility”, and anticipated prisoner exchanges. Indeed, the Qur’an went as far as to suggest that prisoners could be “set free as a favor”.

Going even further back, Deuteronomy 20:19 discusses environmental damage during warfare:

19 “When you besiege a city a long time, to make war against it in order to capture it, you shall not destroy its trees by swinging an axe against them; for you may eat from them, and you shall not cut them down. For is the tree of the field a man, that it should be besieged by you?

A somewhat comprehensive set of laws of war, stemming from tradition and religion, was present in Ancient Greece as well. Adriaan Lanni, in her paper The Laws of War in Ancient Greece mentions:

There was a relatively effective law of war in ancient Greece. But the Greek law of war did not encompass humanitarian ideals. Instead, it focused on protecting sacred objects and observances. The great irony here is that despite the central role played by religion and honor in the Greek laws of war, these laws were indifferent to considerations of mercy and the protection of noncombatants. Notwithstanding Thucydides’ grim view of the efficacy of international law, I will argue that the evidence from ancient Greece actually supports the position that international law did serve as a meaningful check on state behavior.

...

But for the most part the law of war in archaic and classical Greece was informal and customary, arising from shared norms and practices rather than explicit laws and treaties. Our sources generally refer to this body of law with such phrases as the “laws of the Greeks,” “the common laws of the Greeks,” and “the laws common to all men.” The limits on warfare appear to have been relaxed when fighting non-Greek barbarians. But the distinction between “the laws of the Greeks” and “the laws of mankind” is not consistently applied, and it is not clear whether there was a shared understanding of precisely which laws applied only to Greeks and which were wider in scope.

As for a contemporary source, Herodotus in passage 82 of the first book (Κλειώ) of his Ιστορίαι (Histories) mentions bilateral agreements to limit the number of combatants:

It chanced, however, that the Spartans were themselves just at this time engaged in a quarrel with the Argives about a place called Thyrea, which was within the limits of Argolis, but had been seized on by the Lacedaemonians. Indeed, the whole country westward, as far as Cape Malea, belonged once to the Argives, and not only that entire tract upon the mainland, but also Cythera, and the other islands. The Argives collected troops to resist the seizure of Thyrea, but before any battle was fought, the two parties came to terms, and it was agreed that three hundred Spartans and three hundred Argives should meet and fight for the place, which should belong to the nation with whom the victory rested. It was stipulated also that the other troops on each side should return home to their respective countries, and not remain to witness the combat, as there was danger, if the armies stayed, that either the one or the other, on seeing their countrymen undergoing defeat, might hasten to their assistance. These terms being agreed on, the two armies marched off, leaving three hundred picked men on each side to fight for the territory. The battle began, and so equal were the combatants, that at the close of the day, when night put a stop to the fight, of the whole six hundred only three men remained alive, two Argives, Alcanor and Chromius, and a single Spartan, Othryadas. The two Argives, regarding themselves as the victors, hurried to Argos. Othryadas, the Spartan, remained upon the field, and, stripping the bodies of the Argives who had fallen, carried their armour to the Sparta camp. Next day the two armies returned to learn the result. At first they disputed, both parties claiming the victory, the one, because they had the greater number of survivors; the other, because their man remained on the field, and stripped the bodies of the slain, whereas the two men of the other side ran away; but at last they fell from words to blows, and a battle was fought, in which both parties suffered great loss, but at the end the Lacedaemonians gained the victory. Upon this the Argives, who up to that time had worn their hair long, cut it off close, and made a law, to which they attached a curse, binding themselves never more to let their hair grow, and never to allow their women to wear gold, until they should recover Thyrea. At the same time the Lacedaemonians made a law the very reverse of this, namely, to wear their hair long, though they had always before cut it close. Othryadas himself, it is said, the sole survivor of the three hundred, prevented by a sense of shame from returning to Sparta after all his comrades had fallen, laid violent hands upon himself in Thyrea.

Interesting reads:

• A Theory of the Laws of War, Eric A. Posner (PDF)
• The Evolution of Law of War
• Laws of war, on Wikipedia