Why did Churchill become the PM of Britain during WWII instead of Lord Halifax?

Churchill was advocating a hard stand against Germany for many years. So when the war eventually started, he was the most natural candidate. Notice that he was voted out of his position immediately when the war ended. He was considered an optimal leader for war time, and probably he was one.

The person who would replace Chamberlain had to form a National Coalition, and, hence, had to have Labour backing. Labour leadership did not back Halifax because of his prior appeas*m*nt policies, so even though the Labour NEC did not express a preferred PM, Labour would support Churchill over Halifax. In addition, Halifax had both personal ailments and was rather indiscreet which likely helped him decide that he shouldn't take up the position.

I'm basing this answer on McKinstry's 'Attlee and Churchill'.

As Halifax was a member of the previous administration, Attlee and Labour refused to serve under him as Clem Davies reported from a lunch on 9th May where, likely overemphasising his own importance, Davies met with Attlee and Greenwood. After that lunch, Davies went to Bob Boothby and told him that Labour viewed Halifax's administration as a continuation of the Chamberlain government because of Halifax's backing of appeas*m*nt, and would not serve under him. Even if not exactly what Attlee had in mind, this was used by Boothby when talking to other Commons members.

Then, in the afternoon of 9th May, a series of meetings occurred. Firstly, Chamberlain and Halifax convened and Churchill arrived. Attlee and Greenwood were there next, for general discussions of forming a government. In this, the Labour leadership refused to serve under Chamberlain, but couldn't commit without speaking to the NEC. With this statement, the Labour leadership—Attlee and Greenwood—left the meeting. McKinstry's evaluation of all of the memoirs of the people present notes that Halifax thought the Labour leaders 'evasive' and Churchill thought the discussion 'polite'. Greenwood's memoirs had him openly confront Chamberlain compared to the other descriptions of the meeting.

Now, David Margesson, the Chief Whip, joined Chamberlain, Halifax, and Churchill. Churchill said nothing originally to Margesson's comment that Chamberlain has to go, but it is not certain who can succeed him. It is thought that Halifax now opened with his reluctance to lead, recommending Churchill instead. McKinstry summarises this:

Churchill, in his own narrative, also recorded how Halifax listed the hurdles he would face, especially his position in the Lords: 'He spoke for some minutes in this sense, and by the time he had finished it was clear that he duty would fall upon me - had in fact fallen upon me.'

This is, insofar as I can tell by the content, the meeting that is also detailed in @PieterGeerkens' answer. Chamberlain is noted as having supported this narrative both in a summary to Joe Kennedy as well as to Lord Camrose. Halifax's version, offered to his private secretary Charles Peake, differs slightly with an emphasis on Margesson speaking against Halifax before Halifax could say anything. McKinstry also adds that Halifax had been ailed that day by his stomach ache.

Meanwhile, Commons' opinion had been canvassed throughout 9th May by the anti-Halifax Bob Boothby. The Labour conference decided, on 10th May, that they would not serve under Chamberlain while they did not express preference for PM. When Chamberlain got this news, he went to the King, who in turn asked for Halifax to be nominated—but Chamberlain refused in light of the previous discussions, including the strengthening opinion in the Commons.

McKinstry also notes that Halifax's personal reluctance—which is not disputed—is likely to have stemmed from his loose attention to national security affairs as he told his lover of a number of state secrets, and could therefore be taken to task by the House for lack of integrity.

In brief, Churchill became a PM because Lord Halifax did not want the job (at that particular time). The details of exactly why he did not want the job vary a bit from one account to another, but all accounts paint a very consistent picture.

1. There is a lovely description of this story, with references, in the Wikipedia article on Lord Halifax. In particular:

As Lord Beaverbrook said, "Chamberlain wanted Halifax. Labour wanted Halifax. Sinclair wanted Halifax. The Lords wanted Halifax. The King wanted Halifax. And Halifax wanted Halifax." Only the last sentence was incorrect, however; Halifax did not want to become Prime Minister. He believed that Churchill's energy and leadership skills were superior to his own.

The given reference is:

Blake, Robert (1993). "How Churchill Became Prime Minister". In Blake, Robert B.; Louis, William Roger (eds.). Churchill. Oxford: Clarendon Press. pp. 264–270.

Blake's article says:

(The biographer above is The 2nd Earl of Birkenhead and the book is "Halifax: The Life of Lord Halifax", 1965; page 453. The 2nd Earl of Birkenhead served a Parliamentary Private Secretary to Lord Halifax in 1938–39 and as Lord-in-waiting to King George VI in 1938–40, so he also had an insider knowledge of the events.)

2. A slightly different (but consistent with the above) take is given in

Roy Jenkins, "Churchill." London: Pan, 2002.

One important thing: Jenkins uses a variety of sources to support his description of the events (and is often critical of Churchill's own recollections).

In brief, Halifax had chickened out (of the PM job), but wanted to exercise indirect control on Churchill as a PM. The latter was a miscalculation.

This [Churchill's] account is not without a certain central truth, but is wholly inaccurate as to times and participants. The crucial meeting took place not on the Friday morning but on the Thursday afternoon; Margesson was also present; and it required no determination not to break a long silence on Churchill’s part to get Halifax to exclude himself. He had already done so at a 10.15 bilateral meeting with Chamberlain on the Thursday morning. There he stressed the great disadvantage he would suffer as a Prime Minister who was a peer, and for the first time used the phrase that the thought of being so ‘left me with a bad stomach-ache’.

The next day, Rab Butler, who was Halifax’s parliamentary undersecretary and very suspicious of Churchill, tried to get hold of his chief for a last-minute attempt to persuade him to change his mind. Butler was told that Halifax had gone to see his dentist, which suggested that his aches were not confined to his stomach, but also showed a high degree of disengagement. There is no doubt that Halifax was resolved not to become Prime Minister in the circumstances of 9–10 May 1940. Had he wished otherwise there is equally little doubt that he could have secured the appointment.

[The boldface font is mine.]

He was the preferred establishment candidate. The position of the Labour party was equivocal. This was key, for it was Chamberlain’s awareness that they would not serve under him which precipitated his decision to create a vacancy. Dalton was in favour of Halifax and, a little unreliably, reported Attlee as being so too. Morrison inclined that way. But, as Robert Blake has percipiently pointed out, they had ‘not a choice but a veto’. The veto they would not have exercised against Halifax, but nor were they inclined to do so against Churchill.

Throughout those two days of crucial political adjustment there is no doubt that, unlike Halifax, Churchill very much wanted to be Prime Minister. On the Thursday morning immediately following the conclusion of the debate and his late-night interview with Chamberlain he had telephoned Eden and asked him to lunch with him that day. When Eden got there he was surprised to discover a third figure in the shape of Sir Kingsley Wood, then Lord Privy Seal and hitherto a firm Chamberlain loyalist. But Wood had decided that, in Chamberlain’s own interests as well as in those of the country, it was time for him to go, and he was firmly convinced, even before Halifax had contracted his ‘stomach-ache’, that Churchill was the essential successor. His views were not crucial on that Thursday, for Chamberlain and Halifax had already come to that conclusion without Wood’s intervention.

What is more open to doubt is how self-abnegatory were Halifax’s motives. He was not called the ‘Holy Fox’ for nothing. He believed he was wiser if not more energetic or militarily focused than Churchill. He believed he might exercise more control over this impetuous figure (to whom, together with some of his associates such as Beaverbrook and Bracken, he was known to refer as ‘gangsters’) from a strong Cabinet position and with the dignity of having declined the premiership than if he were jostling with him for control of what would have been nominally (but only nominally) a Halifax government.

Furthermore the Churchill government might quickly fail. There were many who thought this at the beginning. In that event he (Halifax) could step in from a position of strength, unthreatened by a powerful rival, and pick up the pieces – although they would probably have been the fragments of an independent Britain as well as those of political power. Both these hypotheses were surmises, although there were a few hints in Halifax’s private correspondence that they had some validity. And both proved false. Over the remainder of 1940, until he was sent as ambassador to Washington, Halifax’s influence within the administration diminished rather than increased, and the Churchill government, so far from being ‘nasty, brutish and short’, survived for five years and became the most famous of the century, maybe indeed of the whole history of British Cabinets. For this we owe much to the fact that Halifax, who on 9 May could have become Prime Minister, wisely declined to do so.

...

Throughout those two days of crucial political adjustment there is no doubt that, unlike Halifax, Churchill very much wanted to be Prime Minister. On the Thursday morning immediately following the conclusion of the debate and his late-night interview with Chamberlain he had telephoned Eden and asked him to lunch with him that day. When Eden got there he was surprised to discover a third figure in the shape of Sir Kingsley Wood, then Lord Privy Seal and hitherto a firm Chamberlain loyalist. But Wood had decided that, in Chamberlain’s own interests as well as in those of the country, it was time for him to go, and he was firmly convinced, even before Halifax had contracted his ‘stomach-ache’, that Churchill was the essential successor. His views were not crucial on that Thursday, for Chamberlain and Halifax had already come to that conclusion without Wood’s intervention. They may have been more so on the next day when, following the early morning news of the German offensive in the West, Chamberlain was tempted to use this as an excuse for staying on ‘until the French battle was finished’. Wood’s signal service then was to tell him, as a friend.”

3. Lastly, the following is taken from page 181 in

Jeremy Havardi, "The Greatest Briton: Essays on Winston Churchill's Life and Political Philosophy", Shepheard-Walwyn; 2010.

What is certain is that Halifax had decided not to accept the top job. The most glittering prize in British politics was there for the taking – but Halifax turned it down. Historians have long speculated on the reasons for this spectacular self-denying ordinance.

In his diary, Halifax wrote the following explanation:

I had no doubt at all in my own mind that for me to take it would create a quite impossible position. Quite apart from Winston’s qualities as compared with my own at this particular juncture, what would in fact be my position? Winston would be running defense, and in this connection one could not but remember how rapidly the position had become impossible between Asquith and Lloyd George, and I should have no access to the House of Commons. The inevitable result would be that being outside both these vital points of contact I should speedily become a more or less honorary Prime Minister, living in a kind of twilight just outside the things that really mattered.

[The italics are mine]

In The Gathering Storm, Churchill describes the key scene with Chamberlain and Lord Halifax thus:

At eleven o'clock I was again summoned to Downing Street by the Prime Minister. There once again I found Lord Halifax. ... [The Prime Minister] told us that he was satisfied that it was beyond his power to form a National Government. ... The question therefore, was whom he should advise the King to send for after his own resignation had been accepted. ... He looked at us both across the table.

I have had many important interviews in my public life, and this was certainly the most important. Usually I talk a great deal, but on this occasion I was silent. ... As I remained silent a very long pause ensued. It certainly seemed longer than the two minutes which one observes in the commemoration of Remembrance Day. Then at length Lord Halifax spoke. He said that he felt that his position as a peer, out of the House of Commons, would make it very difficult for him to discharge the duties of Prime Minister in a war like this. He would be held responsible for everything, but would not have the power to guide the assembly upon whose confidence the life of every government depended. He spoke for some minutes in this sense, and by the time he had finished it was clear that the duty would fall upon me - had fallen upon me.

There being only three - Chamberlain, Halifax, and Churchill - in the room, it's unlikely that any serious dispute about the events that day will arise. It's a (possibly) biased account as well as a first hand one; but it's the best account we have.

For the reasons spoken, in addition doubtless to reasons unspoken, Halifax deferred to Churchill.

In commentary, one notes that on this occasion as well as one a short time earlier, Churchill arrived at Number Ten to find Halifax already there with Chamberlain. It's not impossible to imagine that conversations between Chamberlain and Halifax occurred before Churchill's arrival; and that these two meetings were a test of sorts of Churchill. However I've not seen any explicit description of such. If one does imagine that might have occurred, then one must also accept that, whatever the nature of the imagined test, Churchill passed it; was recommended to the king by Chamberlain with Halifax's blessing; and successfully formed a National Government for the duration of the War.

Update

Notwithstanding the dramatic description above by Churchill, of a meeting between the big three on Friday morning, May 10, Manchester and Jenkins both state a belief that the pivotal meeting described by Churchill occurred on Thursday afternoon, May 9; was attended by Chief Whip Margesson in addition to Chamberlain, Halifax, and Churchill; and that Halifax had already that morning, in a 10:20 am meeting with Chamberlain, expressed his disinclination to accept the premiership.