Context of sign telling Japanese people to "keep moving"

This specific photograph was taken in early May 1923 in Hollywood California. The picture as mentioned in an earlier comment was the result of a local movement calling itself the Hollywood Protective Association. This anti Japanese association was protesting the purchase of a building on Tamarind Avenue they believed was going to be a Japanese church. The association was worried the church would cause an influx of Japanese to move into their community.

Other signs plastered around this Sunset Boulevard neighborhood read "Japs Keep Out, Keep Out Japs, We Don't Want You Here, and This is a White Man's Neighborhood.

This goes back much further than the 1920s. White America was quite racist up until the civil rights battles of the 1960s (and this racism continues to this day). The sole dissenting voice against the Civil Rights Cases and Plessy v. Ferguson, John Marshall Harlan, wrote in his dissent of Plessy v. Ferguson that

In view of the constitution, in the eye of the law, there is in this country no superior, dominant, ruling class of citizens. There is no caste here. Our constitution is color-blind, and neither knows nor tolerates classes among citizens. In respect of civil rights, all citizens are equal before the law.

While that sounds lofty, he also wrote in the same dissent that

There is a race so different from our own that we do not permit those belonging to it to become citizens of the United States. Persons belonging to it are, with few exceptions, absolutely excluded from our country. I allude to the Chinese race.

He wrote this in part because the Chinese Exclusion Act of 1882 had already been law for fourteen years at the time of the Plessy v. Ferguson decision. Harlon used that law and the strong anti-Asian sentiments of the time to illustrate how very bad a precedent the majority was creating in that ruling. However, even Harlan (who was very nondiscriminatory by the standards of his time) could not get over anti-Asian sentiments. He ruled against people of Chinese descent a number of times.

Anti-Asian sentiments continued in America well past World War II. A large number of Korean citizens migrated to the US after the Korean war, and a large number of Vietnamese citizens migrated to the US after the Vietnam war. A large number of those Vietnamese migrated to areas that had conditions similar to home: Hot, humid, and close to water so they could fish. That would be the very, very deep south. Some rather nasty racist incidents arose as a result of that migration.

The above does not directly address the anti-Japanese sentiments expressed in the photo shown in the question. My answer addresses anti-Chinese, anti-Korean, and anti-Vietnamese sentiments. It does not address anti-Thai or anti-Japanese sentiments in the US. And yes, those sentiments did (and still do) exist. Witness the Japanese internment camps during WWII. "They don't look like us" goes back to before humans knew how to write.

During the early 20th century in the US, a strong nativist movement developed in response to increased immigration from Eastern Europe and Asia. The Immigration Act of 1921 and the Johnson-Reed Immigration Act of 1924 limited European immigration to 3% and then 2% of the total US population of the immigrants nation of origin. In the West, laws prevented Japanese immigrants from obtaining citizenship, and initially placed them into segregated schools. This was only ended by Theodore Roosevelt "gentleman's agreement" with Japan, ending Japanese school segregation in the US in exchange for the Japanese government limiting immigration to the US. Despite these law changes, many Western Americans remained adamantly anti-Japanese, due to nativist tensions of immigrants stealing jobs. This was compounded by racial tensions over the darker skin of Asian immigrants.

This was a 1920s era diatribe against "all" Asian people, not a World War II slogan against Japanese. It came during the era of the American eugenics movement, which sought to "purify" the white race. It went further than the 1890s "separate but equal" movements in the American South to separate blacks and whites.

"Purity," in this context, meant trying to avoid all contact with "lesser breeds," not just "separate" development paths. From the Mission statement of the Eugenics Record Office:

"Society must protect itself; as it claims the right to deprive the murderer of his life so it may also annihilate the hideous serpent of hopelessly vicious protoplasm." (Emphasis mine.)

This meant that (some) whites not only didn't want non-whites "living" in their neighborhood, but even having them "pass through" their neighborhood was unacceptable. Regarding Asians, "keep moving," or "better" yet, "keep out," was the only acceptable posture to these people.

The eugenicists considered "colored" (Asian or black) people an existential threat to their society. This was reflected in the anti-immigration laws of the time, e.g, the National Origins Act of 1924 that placed annual quotas on immigration from each "nation" of 2% of its 1890 U.S. population, which was primarily directed against Asians.

The context for this specific photo was that a number of Japanese-Americans had been looking for housing in Hollywood, where they worked in menial positions. In response, white residents formed something called the "Hollywood Protective Association" in 1923 and campaigned to "keep Hollywood white".¹ This is a photo of a member of that organisation, pointing at a typical sign of the era.

This was part of a broader trend of anti-Japanese racism and xenophobia in early 20th century West Coast, which is very well documented. For example, the Times warned that Japanese "control of California farm lands" was "endangering white supremacy in California".²

I'm not sure there's any need to look too deeply into motivations. Prejudice against the foreign and the dissimilar are not particularly uncommon attitudes in human society. Japanese immigrants, who began arriving in continental United States in the late 1800s, were just another Asian ethnicity facing discrimination in contemporary California. The Chinese were categorically banned in 1882 for instance, which actually created the demand for Japanese migrant labour in the first place.

Something not really appreciated neither then nor since is the antipathy such treatment evoked back in Japan.

References:

1. Kurashige, Scott. The Shifting Grounds of Race: Black and Japanese Americans in the Making of Multiethnic Los Angeles. Princeton University Press, 2010.
2. Charlotte Brooks. "Alien Neighbors, Foreign Friends: Asian Americans, Housing, and the Transformation of Urban California .Chicago: University of Chicago Press, 2009.