Why is the heir to the British throne called "Prince of Wales"?

(I know Lars Bosteen has provided an answer but in the interest of completing this answer, I've provided a slight update.)

This is two questions rolled into one:

1. Why, in particular, Wales?
2. How did the convention (or tradition) develop for appointing the heir apparent the title: Prince of Wales

Why Wales?

The principalities of Wales was not unified under a single banner until Edward I (Edward Longshanks), King of England, executed the last native Prince of Wales, Dafydd ap Gruffydd (David III) in 1283.

Edward I created the institution of Prince of Wales in 1301, at a parliament at Lincoln (not Westminster). Edward of Carnarvon (Edward II) was the first English Prince of Wales.

As the conqueror, Edward I used this opportunity to deny the Welsh their right to determine the next heir-apparent, in particular, the Edling (Welsh: etifedd). As for determining their heir-apparent, the Welsh had a tradition of choosing their leader and the title Edling is likely borrowed from the Anglo-Saxon, Ætheling (also spelt Aetheling, Atheling or Etheling).

(Update: More info on Welsh tradition on inheritance and the history of Edward's fight with the Welsh.)

Welsh Inheritance Tradition

In terms of Welsh tradition on inheritance, it was unique. The Welsh practiced partibility, as opposed to primogeniture (i.e. inheritance going to the eldest son). This created all kinds of in-fighting between relations after the passing of Welsh leaders and Edward's father, the less-than-exemplary Henry III used it to great effect after the passing of Llywelyn the Great in 1240.

More perplexing still for English and Scottish onlookers, and far more problematic, were Welsh social attitudes, which stood in sharp opposition to their own. Take, for instance, the rules governing inheritance. In England and Scotland, and indeed almost everywhere else in western Europe, the rule was primogeniture: firstborn sons inherited estates in their entirety. This was hard on any younger brothers or sisters, but had the great advantage of keeping a family’s lands intact from one generation to the next. In Wales, by contrast, the rule was ‘partibility’: every male member of the family – not just sons and brothers, but uncles and nephews too – expected his portion of the spoils, and rules of precedence were only loosely defined. This meant that the death of a Welsh landowner was almost always followed by a violent, sometimes fratricidal struggle, as each male kinsman strove to claim the lion’s share.

The result of this idiosyncratic approach to inheritance was that Welsh politics were wont to be tumultuous. The fact that partibility applied at the highest levels was one of the main reasons why there was no single political authority in Wales as there was in England and Scotland. Welsh poets spoke of their country as if it were neatly divided into three kingdoms, but this was a broad simplification; the reality was a complex patchwork of petty lordships. Occasionally one ruler might, through force of arms, diplomacy or sheer good luck, contrive to establish something greater. But such constructs were always temporary. When a successful Welsh ruler died, his work was swiftly undone by the general carve-up that inevitably followed.

Source: Marc Morris, "A Great and Terrible King: Edward I and the Forging of Britain" (Windmill, 2009)

So, by Welsh convention, if there was no Edling, their partible inheritance system would have caused another round of division and in-fighting for leadership.

Conquering Wales: 1256 - 1283

(there's a map for reference below)

Finally, if the above was not a good enough reason, the history of why and how the title of Prince of Wales came to be recognised is also relevant. And the key factor why this title was so coveted is simply because it was extremely hard fought - nearly 30 years:

1. It started in 1256, before Edward I started his reign in 1272 and was finally achieved nearly 30 years later, when he was King of England.
2. The resurgence of the Gwynedd, led by Llywelyn ap Gruffudd (Llywelyn II, grandson of Llywelyn the Great) in late 1256 was in fact caused by Edward's own visit to Wales earlier in the year, when the king's chief steward, Geoffrey de Langley, ".. boasted before the king and queen that he had all the Welsh in his grip"
3. The effect of this boast was a rebellion, and with help from Llywelyn ap Gruffudd, was a full-scale military incursion into English lands as well that of Marcher lords. In addition, Edward's own castles at Dyserth & Deganwy were besieged (newly built by his father, Henry III, but given to Edward as endowment for his marriage to Eleanor of Castile). This is important because it was not Henry III's fight -- most of the lands in Wales held by Henry were already bestowed to Edward.

The Welsh inhabitants were finding English rule in Perfeddwlad oppressive. Geoffrey de Langley, the representative of the English king in Wales,‘boasted before the king and queen that he had all the Welsh in his grip’. The response was explosive. ‘The Welsh, coming out of their own territory, gathered a great army, headed by Llywelyn II, a handsome man and vigorous in war, who had, as it were, collected together all the Welsh to himself.’ Within a week Perfeddwlad was in Llywelyn’s hands. Over the next decade his series of military and political successes was continuous. In campaigns reminiscent of those of his grandfather, he expelled Gruffudd ap Gwenwynwyn of south Powys, raided as far south as Pembrokeshire and made a succession of conquests on the Middle March at the expense of Marchers like the Mortimers,taking Builth and Brecon.

Source: The New Cambridge Medieval History: Volume 5, c.1198-c.1300, p.813

4. For decades after this, Edward and his father tried, but failed to subdue Llywelyn II.
5. After the inital attempts in 1257/8, and because he was not yet king, he needed further assistance from his father which was not forthcoming. He turned for help from the lords of the March of Wales and others, one of which was Richard, 1st Earl of Cornwall (see below on Duchy of Cornwall). Note that Richard of Cornwall, was at the time one of the wealthiest man of all Europe. (Mark C. Wallace has a short, but interesting, paragraph on the Marcher lords here)
6. Henry III, as King, had to recognise Llywelyn II as Prince of Wales in 1267. Not even Llywelyn the Great achieved this feat of forcing the King of England to recognise a ruler of Gwynedd over Wales.

The Peace of Montgomery between Llywelyn and the English crown, made in 1267, ceded to the prince virtually all his conquests,the title ‘prince of Wales’and the fealty and homage ‘of all the Welsh barons of Wales, so that those barons shall hold their lands in chief from the prince and his heirs’. The princely dynasty of Gwynedd was thus recognised as the sole channel linking the native Welsh chiefs and the English crown.

Source: The New Cambridge Medieval History: Volume 5, c.1198-c.1300, p.813

7. Edward's reign then started in 1272 and he finally conquered Wales with passing of David III (younger brother of Llywelyn II and the the last native Prince of Wales) in 1283.

To end, this map of Wales after the treaty might be useful:

• green - Gwynedd, Llywelyn ap Gruffudd's principality
• purple - Territories conquered by Llywelyn ap Gruffudd
• blue - Territories of Llywelyn's vassals
• orange - Lordships of the Marcher barons
• yellow - Lordships of the King of England

The Convention - Prince of Wales, Duchy of Cornwall

The Duchy of Cornwall was created by royal charter in 1337 by Edward III for the Black Prince. Both Edward III and the Black Prince were excellent role-models (of the time) and this is probably why this practice became entrenched, and developed into English parliamentary convention.

The role of the duchy is to provide revenue for the heir-apparent (assets are held in trust). The Duchy of Cornwall Act, 1860, organised and standardised the governance of the duchy and the holdings and perquisites are actually located beyond the modern county of Cornwall, to Devon, Somerset & Gloucestershire (closest to Wales).

A recent discussion on the Rights of the Sovereign and the Duchy of Cornwall Bill, Hansard, 2013.

The title of Duke of Cornwall is granted to the eldest son, but not necessarily the heir-apparent. More info available at their website.