Why did LBJ, a staunch segregationist, champion and sign the 1964 Civil Rights Bill?

I think a part of the reason is that for his career in the House and then the Senate, to be elected he needed first the votes of the people in the Texas Hill country and then the the entire state of Texas. One needed to be a racist to get elected. After he became Vice President and especially after becoming President, the voters he needed were from the entire US. The US was moving away from its inhumane treatment of it's black citizens and the South's barbaric Jim Crow laws.

As for the assistant and the dog, I have no idea if the story is true or not. But, it does illustrate a very good point. For a black American to travel from Texas to Washington D.C. in those days required substantial planning and could be extremely dangerous. The chances are very good that you would be exhausted when you arrive.

In the late 1950's Senate Majority Leader Lyndon Baines Johnson was forming his first presidential campaign. He was advised his largest liabilities for national office was his vocal and consistent leadership in defense of segregation. This all but made him unelectable in the North and the West. In 1957 Johnson could not afford to alienate any of these regions. Johnson would try to finesse this liability. LBJ would outwardly support the 1957 civil rights bill, meanwhile he used his power as Senate Majority Leader to divert the bill into a committee controlled by a powerful southern committee Chairman. In the committee the 1957 bill would be neutered.

Johnson sent the (1957 civil rights) bill to the judiciary committee, led by Senator James Eastland of Mississippi, who proceeded to drastically alter the bill.

In 1957, Johnson hoped that he would both be given credit for passing the bill by the bills northern supporters, and would be given credit for weakening the bill from southerners opposed to the legislature. Legislatively Johnson was successful, politically it was a failure. The weakened bill passed. However, Johnson lost the Democratic nomination for the Presidency in 1960 to the junior Senator from Massachusetts, John F. Kennedy who had a much bolder agenda on civil rights.

As @congusbongus stated above in his answer...Lyndon Johnson was first an ambitious politician who long craved to become President. As a Freshman congressman in the 1930's he told his staffers to refer to him as LBJ because it sounded like FDR. "FDR-LBJ, LBJ-FDR "get it?". Johnson had arrived in Washington planning on becoming president. (From Master of the Senate, page 100)

In preparation for the 1964 election Johnson would prove to be one of the boldest and most effective desegregationists when it was politically in his interest to be so. On November 22, 1963, President Kennedy was shot and killed in Dallas Texas; leaving LBJ President. LBJ inherited a powerful political machine and an American public who mistrusted him due to his well known and lengthy resume as leader in the Senate. Johnson took immediate steps to close the daylight between himself and the fallen president.

The fallen President who had tried and failed to pass what would become the 1964 civil rights bill. Now with the entire nation mourning, it was in Johnson't political interest to do what many thought was impossible. Pass this landmark legislation and broaden his own political base by showing he was a man of vision capable of leading the agenda which Kennedy had presented and sold to the nation. Johnson would resurrect and pass the most comprehensive civil rights bill in the nations history.

**Robert Caro Pulitzer Prize-winning biographer and historian on Lyndon Baines Johnson **

from Dian Rehm show
CARO You know, Johnson was a very complicated man. He was filled with ambition and he was filled with compassion, but the truth is, whenever the two of them collided, whenever he had to be one the Southern side, ambition was what won. But I wrote "when ambition and compassion were both pointing in the same direction, he was a force that was unlike anything else in American history."

REHM All right. But then, what do we know about why he decided to move so quickly once he became president, ( and target the 1964 civil rights bill)? 11:14:18

CARO Well, I'll tell you what he said to someone who doubted he was sincere. A speech writer of President Kennedy's before him, named Richard Goodwin, quite a brilliant man, Goodwin sort of asked the same question you did. And Johnson said, you know -- why are you making this a priority? And Johnson said, you know, when I was teaching those kids, I swore that if I ever had the opportunity to help them, I would.

President Johnson in 1963 still had never won a national election. He had never even competed in a national election as head of the ticket. Now he was President, Leader of the political machine which Kennedy had built; only the Kennedy's hated him. So five days after President Kennedy was shot; on Nov 27, 1963; as the entire country mourned, on national TV, in his first address to a joint session of congress; President Johnson ties the legacy of the dead president and the 1964 civil rights bill to his own campaign for President.

Johnson told the legislators, "No memorial oration or eulogy could more eloquently honor President Kennedy's memory than the earliest possible passage of the civil rights bill for which he fought so long.... Then he told his fellow southerners, "We have talked long enough in this country about equal rights. We have talked for one hundred years or more. It is time now to write the next chapter, and to write it in the books of law."

Johnson then goes on to champion that 1964 civil rights bill and in July of 1964 as President he signed the landmark legislation. Tying himself and his election to the fallen president and securing one of the largest landslide victories in the nations history. With Barry Goldwater only winning 6 states. His home state of Arizona; and the 5 deep south states of Louisiana, Alabama, Mississippi, Georgia, and South Carolina.

Johnson went on after the 1964 Civil Rights Act, to pass the 1965 Voter's Rights Act. Together these two landmark Acts ushered in dramatic changes the country.

This is just to add to Pieter's excellent answer.

Some executives go through a "shark to dolphin" transition as they rise. Lyndon Johnson more than most. That is, they lose some of their rough edges as they get more and more responsibility and take on larger roles. Instead, they become more "statesmanlike."

That is to say that Johnson started out as a "hometown boy" as a local politician from the narrowest of rural backgrounds. Over time, he made a larger rise than most, to the national, and then international, stage. Unlike e.g Robert E. Lee, whose views never rose above the "state" level, Johnson "grew" into each new level, if not when he reached it, then sometime after. His earlier segregationist tendencies reflected his hometown views, and his civil rights tendencies reflected his national views.

This transition was best described in the multi volume series, Means of Ascent.

There is not much of a reason to assume that LBJ wasn't racist. It may be easier to forget in our age but racism like sexism and other forms of chauvinism is a natural urge in the context of humans' tribal history that needs to be educated out of civilized man in order to make him suitable for being a member of a global society.

This is a process taking generations, and it has constant setbacks of the "told you so" variety of semi-selffulfilling prophesies. This means that those in the position to make most progress are starting from being behind most.

The time to go through with racial emancipation had arrived, the Supreme Court had started setting some cornerstones establishing equal rights and pronouncing institutional cases of segregation as unconstitutional (cf Brown vs Board of Education in 1954). Eisenhower, a Republican president, had enforced the desegregation of education in Arkansas by putting the National Guard under Federal control and sending an army division for protecting black students, steps so uncompromising and radical that all of the leverage of a white Republican U.S. Army general was wanted in letting the U.S. change the course of its history for good.

As a Democrat, LBJ would have looked bad dialling back a movement for civil rights championed by a Republican president taking his oath of office seriously.

So he decided to make history by working with rather than against the times. It would have been stupid to sacrifice what Eisenhower had wrested from the nation when the respective topics were actually more likely to benefit the Democratic party than the Republican one in the long run (never mind that the Republican Lincoln fought the Civil War on the ticket of racial emancipation).

Current voter profiles along party lines show that LBJ did the Democratic party a long and defining service by taking ownership over the topic of civil rights.

Why? Because from an egocentric point of view, LBJ became aware of situations that affected him and his (extended) "family". Please allow me to relate a specific anecdote to illustrate that LBJ's large ego (well-known, cited here: https://www.theatlantic.com/magazine/archive/1998/04/three-new-revelations-about-lbj/377094/) was insulted by this everyday racism against his own chauffeur/manservant.

My father-in-law, who graduated from the LBJ Law School at University of Texas at Austin in the early 70s, related an anecdote to me while we were visiting the Lady Bird Johnson Wildflower Garden.

The Johnsons had a black assistant/chauffeur/manservant. Eventually, this guy (not sure if it was Robert Parker or Carroll Kreach) had to deliver the Johnson's pet dogs to Washington, D.C. The Johnson's loved dogs and made their beagles "Him" and "Her" famous.

LBJ was disappointed that he showed-up in Washington looking tired as hell, and demanded to know what happened. As the assistant travelled across the southern United States, he was not allowed to rent a motel room for himself. But the dogs, of course, could stay all night.

This particular injustice struck the Johnsons personally. Their stature and good name couldn't prevent it. If anything, this supports @Pieter Geerkens answer that LBJ confronted his own autopilot, lazy racism. I assert that, in part, it happened because of someone close to him.

Maybe when Johnson said “it is not just Negroes but all of us, who must overcome the crippling legacy of bigotry,” he really meant all of us, including himself. --Adam Serwer

LBJ didn't flip. He was racist and only use the Civil Rights Act to further his power and his party's political power. LBJ said to Richard Russell, a fellow Democratic Senator from Georgia:

These Negroes, they're getting pretty uppity these days and that's a problem for us since they've got something now they never had before, the political pull to back up their uppityness. Now we've got to do something about this, we've got to give them a little something, just enough to quiet them down, not enough to make a difference.

Here is the quote from the book Lyndon Johnson and the American Dream:

Most important was the fact that the civil rights issue intensified and brought to focus Johnson’s recurrent fear that “the whole thing”—his leadership, the Senate, the world—would fall apart if he lost control even for a moment, thus permitting the forces of violent division to “get loose.” “One real slip and we’re done for,” he would say again and again, as if both his power and the future of America were fragilely suspended by a gossamer thread. Fearing that the issues of the civil rights question would be “taken over” by the “extremists”—denned as a choice between the irreconcilable views of Southern segregationists and Northern liberals—Johnson felt “driven” to seek a middle course, a legislative formula, that would represent some real progress—enough to moderate liberal passions, but not so unacceptable that it would provoke an open break with the party and its leadership. “I knew,” he later said, “that if I failed to produce on this one, my leadership would be broken into a hundred pieces; everything I had built up over the years would be completely undone.” 27

Less significant than the revelation of personal fear is the fact that Johnson exhibited the prescience to recognize that this issue had dimensions far greater than the difficulties of formulating practicable legislation. He seemed to understand that the issue of civil rights had created a crisis of legitimacy for both the Senate and the Democratic Party. Perhaps it was this understanding that helped Johnson not only to surmount his fears during this struggle but to transform them into instruments of leadership— influencing the action of others by persuading them to share in his apprehension of dangerous possibilities.

Johnson determined that his first task must be to persuade the “reasonable” Southerners to abandon their support for a filibuster, by demonstrating that even if it was successful the only result would be a Pyrrhic victory for the South. Northern passions were rising, becoming “hysterical,” and would no longer accept defeat by filibuster; instead, the attack would focus on the filibuster rule itself. He began with Russell: “These Negroes, they’re getting pretty uppity these days and that’s a problem for us since they’ve got something now they never had before, the political pull to back up their uppityness. Now we’ve got to do something about this, we’ve got to give them a little something, just enough to quiet them down, not enough to make a difference. For if we don’t move at all, then their allies will line up against us and there’ll be no way of stopping them, we’ll lose the filibuster and there’ll be no way of putting a brake on all sorts of wild legislation. It’ll be Reconstruction all over again.” 28

LBJ was a complex figure. He is described as having "uncommon ambition", easily discarding considerations such as ideology for the sake of advancing his career. As a Texas congressman and then Senator, his voting in favour of segregation is completely in line with the political climate of the time and place. Seizing on the nation's grief and support for completing JFK's legacy, he may have seen the Civil Rights act as a political tool to shore up support for his own agenda.

And yet he has long been motivated, probably by his deep religiosity, to help the disadvantaged, such as the poor, but also including African Americans. This aspect he shared with his mentor FDR. For example, as president he launched a series of legislation under the War on Poverty moniker. It's possible that his support for Civil Rights fell along these lines.

But it's important to note that he was most likely racist. He would sometimes refer to the Civil Rights bill as the "nigger bill". He was a product of his time and place. Nevertheless, it's hard to deny that the Civil Rights Act would not have passed if not for his tremendous efforts. Perhaps the best anecdote that illustrates his hypocrisy between ambition and compassion: after the passage of the act, his main concern was that he "delivered the South to the Republican party".

"Why?" questions are particularly difficult to answer. It is not possible to stare into Johnson's soul, searching for motivation, from our vantage point more than fifty years later; instead we are left attempting to analyze his recorded acts and words, leveraging our understanding of human nature.

However, I believe it is fundamental to recognize that racism acts on two levels in everyone: an emotional level and an intellectual level. At the emotional level racism develops from two deep aspects of human psyche: tendencies to "oppose the unfamiliar" and to "support the group". All of us encounter these tendencies; but they are merely tendencies. As humans we have the intellectual capability to set aside our emotional tendencies for rational reasons.

Scant few of us may have been born in situations where those of a different race are a natural everyday phenomenon. But we are all born with an innate sense of group, starting with the family and successively moving out through friends, neighbours, townsfolk, and eventually, in some, race. When an emotional tendency to favour friends and family manifests itself in word and deed, uninhibited by rationality, that is merely nepotism. When it spreads to a blanket preference for one's race, buttressed by rationalization and a dislike of the unfamiliar, then that can truly be labelled a deep racism.

But in between these I see an everyday racism, an autopilot racism if you will, where the emotional tendencies are simply allowed to wash over one without any attempt made to engage the intellect. It is the "lazy human's racism", of simply going with the flow. It seems to me that Johnson's racism was of this type. As a young ambitious legislator raised in a deeply racist and segregated South, "fighting the good fight" might have made him a better human; but would certainly have sunk any aspiration to a political career. Johnson was no Don Quixote, tilting at windmills; he had bigger plans.

Then in the mid-50's, while Johnson is Majority Leader in the Senate, segregation hits the national consciousness. His long-standing support for the poor and downtrodden suddenly coincides with a recognition that many of those are not only black; but are poor and downtrodden because of being black. At that moment the driven, ambitious, and yes, racist, Senator from Texas steps up to the challenge; confronts one of his own demons; and conquers it. He remakes himself as a Civil Rights Champion.

It seems appropriate to note here that, as President, Johnson was uniquely qualified in a manner that few of his predecessors, and none of his successors to date, have been: as a former Majority Leader of the Senate, Johnson understood the legislative process far better than most presidents. Johnson had perfected that craft in what his contemporaries termed "the full Johnson" (or alternatively "the Treatment"):

flattery, deals, pressure, threats — delivered at all hours in manners ranging from lofty to crude

Further

As Richard Russell, the South’s leader in the Senate during the 1960s, put it to a friend a few days after Kennedy’s assassination: “You know, we could have beaten John Kennedy on civil rights, but not Lyndon Johnson.”

So: "Why did Johnson champion the Civil Rights movement from 1957 on?"

Because it was the defining issue of his time; and, to be recognized as the great man that he aspired to be and saw himself as, one had to not only be on the right side; one had to be a champion of the right side.