Why does Northern Ireland not include the counties Donegal, Monaghan and Cavan?

At the risk of being pedantic, it is worth noting that some parts of the three counties traditionally included in the province of Ulster were transferred to the newly created Northern Ireland. However, the overall thrust of your question is correct - the traditional province of Ulster was divided by the Partition of Ireland.

The shape of modern Northern Ireland is the result of deliberate gerrymandering on the part of the British government. The goal was to create a state with artificial borders which included the maximum possible amount of space while also ensuring a strong Protestant majority population.

There was no way to achieve this goal with the inclusion of the entire traditional province of Ulster, so the three counties with a majority Catholic population were omitted. The historical borders of the remaining six counties were also altered slightly so as to exclude as many Catholics as possible.

The Government of Ireland Act of 1920:

In 1920, the Irish War of Independence was raging. The British government had almost completely lost control of the country, and the Provisional Government under Sinn Fein was the de facto government in most of the southern 26 counties. In a panic, the British enacted a new law, the Government of Ireland Act of 1920. The law was ridiculously ambitious and unrealistic. The idea was to split the island into two separate political entities, each with its own parliament, but both still firmly under British oversight.

The unspoken objective was to ensure that Northern Ireland was as large, as economically viable, and as populous as possible, while also ensuring that the population was majority Protestant. This was a significant concern for all sorts of reasons, going back a thousand years, and far too complicated to be repeated here. Suffice to say that Protestants and Catholics in Ireland don't have the best track record in terms of not murdering each other.

The boundary between "Northern Ireland" and "Southern Ireland" was stipulated in the law:

For the purposes of this Act, Northern Ireland shall consist of the parliamentary counties of Antrim, Armagh, Down, Fermanagh, Londonderry and Tyrone, and the parliamentary boroughs of Belfast and Londonderry, and Southern Ireland shall consist of so much of Ireland as is not comprised within the said parliamentary counties and boroughs.

The Anglo-Irish Treaty and Ensuing Agreements:

When the Irish War of Independence (also known as The Black and Tan War) ended in 1921, the peace treaty included provisions for redrawing the border between Ulster and what was then called the Irish Free State. The most important provision in this regard was contained in Article 12:

...a Commission consisting of three persons, one to be appointed by the Government of the Irish Free State, one to be appointed by the Government of Northern Ireland, and one who shall be Chairman to be appointed by the British Government shall determine in accordance with the wishes of the inhabitants, so far as may be compatible with economic and geographic conditions the boundaries between Northern Ireland and the rest of Ireland, and for the purposes of the Government of Ireland Act, 1920, and of this instrument, the boundary of Northern Ireland shall be such as may be determined by such Commission.

Collins and his colleagues demanded that any decision on the border issue must be open for renegotiation at any time after the initial agreement, and it is clear that Collins had every intention of returning to the subject in the near future, and making drastic revisions. Unfortunately for his countrymen, he never got the opportunity to do so.

Shortly afterwards, an additional agreement was reached by the three governments of Britain, the Irish Free State, and the newly formed Northern Ireland. This deal is known as the Collins-Craig agreement, after the principle negotiators, Irish Free State General and Provisional Government Chairman Michael Collins and Ulster Unionist leader Sir James Craig. It was countersigned on behalf of the British government by Winston Churchill. The agreement stated the following:

(7) During the month immediately following the passing into law of the Bill confirming the constitution of the Free State (being the month within which the Northern Parliament is to exercise its option) and before any address in accordance with Article 12 of the Treaty is presented, there shall be a further meeting between the signatories to this agreement with a view to ascertaining:

a. Whether means can be devised to secure the unity of Ireland. b. Failing this, whether agreement can be arrived at on the boundary question otherwise than by recourse to the Boundary Commission outlined in Article 12 of the Treaty.

Essentially, Collins and Craig were trying to find a way to draw the boundaries between Northern Ireland and the Irish Free State themselves, with as little interference as possible from outside parties. The agreement was signed in March of 1922, but relations between Collins and Craig quickly deteriorated, and the agreement itself was soon abandoned. Only three and a half months later, Collins was assassinated on a country road near his childhood home, permanently altering the course of Irish history.

The simmering discontent among the hardliners in the Irish Republican Army, who had opposed the Anglo-Irish Treaty, had led to Collins' death. The hardliners accused Collins of treason and treachery, and had come to see him as a traitor who had sold out his country and his ideals for the sake of personal power and glory. They didn't realize that he was the only thing standing between the Free State Army, which he commanded, and an all out assault on the hardliners. With Collins dead, and his men understandably enraged at his death, the hardliners now found themselves under siege, attacked at every turn, and being ruthlessly pursued and destroyed with extreme prejudice. The Civil War only lasted for about two years, but it claimed the lives of more people than had been killed in the much longer War of Independence.

As a result of the Civil War, the Boundary Commission was unable to do its job until the end of hostilities in 1924. Then, the business of drawing the new map began; the commission included one member from each of the interested parties:

• Justice Richard Feetham for Britain

• Minister of Education Eoin MacNeill for the Irish Free State

• Joseph Fisher for Northern Ireland

The nationalist interpretation of Article 12 was that the Commission should redraw the border according to local nationalist or unionist majorities at the finely granular District Electoral Division (DED) level. Since the 1920 local elections in Ireland had resulted in outright nationalist majorities in County Fermanagh, County Tyrone, the City of Derry and in many District Electoral Divisions of County Armagh and County Londonderry (all north and east of the "interim" border), this might well have left Northern Ireland unviable. Unionists were content to leave the border unchanged.

The Boundary Commission met in 1925 to finalize the negotiations in secret, but a newspaper journalist leaked information about the talks.

On 7 November 1925 an English Conservative newspaper, The Morning Post, published leaked notes of the negotiations, including a draft map. The overall effects of the Boundary Commission's recommendations would have been the transfer of 286 square miles to the Free State and and 77 square miles to Northern Ireland. The leaked report included, accurately, the Boundary Commission recommendation that parts of east Donegal would be transferred to Northern Ireland. Only 1 in every 25 Northern Irish Catholics would have been placed under Free State rule. The Boundary Commission's recommendations would have shortened the border by 51 miles (approx. 18%).

The Boundary Commission's recommendations, as reported on in the The Morning Post were seen as an embarrassment in Dublin. There they were pereceived as being contrary to the overarching purpose of the Commission, which they considered was to award the more Nationalist parts of Northern Ireland to the Free State. Professor MacNeill resigned his cabinet post on 20 November. Despite resigning, he then voted in favour of the settlement on 10 December. It is likely that the press leak caused the boundary negotiations to be swept into the wider agreement signed on 3 December.

_{Maps showing the demographics of Northern Ireland before partition and the changes proposed but scrapped in 1925}

Later that year, negotiations were revived, and the first subject of controversy was the interpretation of Article 12 of the Anglo-Irish Treaty:

The Irish view was that it was only intended to award areas within the six counties of Northern Ireland to the Free State.

The British view was that the entire 1920 boundary was adjustable in either direction, as the Irish side had insisted in the 1921 Treaty that Northern Ireland was deemed part of Ireland until it voted to secede in December 1922, but that the net balance of property and people transferred either way would benefit the Free State.

The Irish still wanted a more equitable arrangement, but the British had all the political leverage. One of the other provisions of the Anglo-Irish Treaty, Article 5, had stipulated that the Irish Free State would be responsible for paying a share of Britain's debts related to the war, but the Free State had no way to pay it. Therefore, the new head of the Irish negotiating team, Collins' successor as Chairman of the Provisional Government, William Cosgrave, cut a deal. Ireland would be absolved of the responsibility of paying the debt, and the borders established in 1920 would remain unchanged. It is possible that Cosgrave also requested that the terms of the agreement not be published.

Opinion among Protestants was uniformly in favor of gerrymandering the border:

A solid Protestant Ulster will be a prop in Ireland to the Empire without which the whole Naval strength of England would be jeopardized. A Protestant and Loyal Ulster would be an invaluable jumping-off point for the British Navy and Army if it were found necessary to use them in case of serious trouble in Ireland or elsewhere. This is sufficient justification for supporting the six county policy. The Empire should count for something.

It is Ulster’s duty, on this score alone, to see that whatever is left of Ulster must be dominantly Protestant, for the safety of the Empire, even though one county only remained.

I consider that by voting for the six counties I have kept my Covenant both in spirit and in letter. My one object in signing the Covenant was to keep Ulster Protestant, and free from any possibility of becoming a part of a Home Rule Ireland with one Parliament in Dublin.

If I had voted for the nine counties I would have been going against both the spirit and letter of the Covenant. Take for example, a ship that has struck a rock and is sinking. The last lifeboat is pushing off with men and women and children. It is so dangerously full that there is no more room. Several people on the wreck jump over-board, swim for the lifeboat and try to scramble into it, with the result that it begins to sink. If they get into the boat they will go down just as surely as if they had stayed on the wreck, and they will have drowned the lifeboat load of passengers who would otherwise have had their lives saved.

For the six counties to jump into an Irish Parliament in Dublin and drown in it with the other three may look heroic, but it would be disastrous to all nine of the counties.

If, however, six strong Unionist Protestant counties hold together on the firm pier of a Protestant Ulster Parliament they will be able to help their brother Unionists in the three counties when these need assistance far better than if all nine were in a hopeless minority in an Irish Parliament, as they undoubtedly would be.

There are 890,880 Protestants in the whole of the nine counties of Ulster. There are 70,510 Protestants and 260,655 Roman Catholics in the three counties. I cannot believe the Protestants in the three counties are willing to swamp 820,370 Protestants merely for the satisfaction of knowing they are all going down to disaster in the same boat.
- Why I Voted for the Six Counties, a leaflet issued in April 1920 by Major Fred Crawford, the former Ulster gunrunner

The British MP Captain Charles Craig, brother of Sir James, openly stated in the House of Commons that a six county Ulster was an absolute necessity because it represented the largest possible area which could be safely held by a Protestant majority population (I'm still looking for the actual quote, and I'll add it when it turns up)