Did the Olympic Comittee consider cancelling the 1936 games in protest?

Why should the IOC cancel the games because someone 85 years later doesn't like Hitler much? And yes, a lot of people don't like him now, but at the time that wasn't the case. He was very popular, including outside Germany. And still that's irrelevant, the Olympics aren't supposed to be about politics.

There were Olympic games much more recently in countries with pretty bad human rights records. Moscow comes to mind, and Beijing.

And if you don't hold the games because someone doesn't like the country they're to be held in, there won't be any games. North Korea objected to the games in Seoul, the USSR to the games in LA, I'm sure someone didn't like that Japan got to organise the games not once but twice (or was it 3 times by now), the list goes on and on.

Pretty much the only reason to cancel the games is if the country organising them is an active war zone, as that would make it way too dangerous for the athletes and audience to go there (if they could get there at all), and would hint at the IOC taking sides in that war which they wouldn't want to do.

The games are about uniting people, not tearing them apart, a place where athletes from countries that are otherwise at odds can come together and compete in a reasonably friendly atmosphere without too much interference by outside agencies.

And that's exactly what happened in 1936. If the German politicians weren't keen on acknowledging the successes of foreigners, that's no problem, the athletes among themselves did acknowledge them.

Yes, there were "considerations" of such a move, cancelation of the games, relocation of the games, or boycotting the games.

In 1933 the whole IOC session in Vienna was dominated by answering this question. While they considered a range of options, we know they decided against such 'harsh' measures and were content with nazi assurances.

The official publication by the IOC reporting on the Vienna session leaves out most of the controversy it is meant to describe but still retains the American position of threatening non-participation if negotiated guarantees should not be met.

These considerations began immediately after the Germans started implementing fascist policies in 1933 and continued throughout until 1936. Although not by the IOC as institution, threatening 'ideas' of individual and high-level members continued to be an issue, one that was lost.

VIENNA, June 3 -- As delegates to the International Olympic Committee meeting, scheduled for next week, gathered today, there was a distinct movement afoot to cancel the award of the 1936 games to Berlin because of the anti-Semitic movement there.
_{–– "Proposal To Shift Olympics Growing; Distinct Move Is Afoot Among International Delegates to Cancel Berlin Award.", NYT, The Associated Press. June 4, 1933.}

BERLIN, Aug. 6. -- An authoritative source expressed the belief today that fear of losing the 1936 Olympic Games because of the campaign against Jews and "political Catholicism" lay behind Nazi attempts to obstruct foreign correspondents.
_{–– "Nazis' Dread of Losing Olympics Is Blamed For Efforts to Ban News of Drive on Foes", NYT, The Associated Press, Aug. 7, 1935.}

The desires and options were for a time quite looking to be successful, but as we know the leave-it-be faction won.

Why was that the case?

The strongest voices against "Nazi Games" were in France, fuelled by German immigrants who fled Nazsim and disliked the outlook of giving Germany an opportunity for propaganda, and in America voices who wanted mainly either to show that 'the American system' was superior to German fascism, or equal participation opportunities for Jewish and black athletes.

After all, Nazi policy is pretty much against the 'spirit of Olympia', something most often argued in France. And this concern led to the open debate from the German side, which wanted to extend racial policy immediately to the games. They were publishing strongly voiced in the Völkischer Beobachter and even worse outlets, how it should be outright forbidden for Jews and black to even show up on the one hand. The named newspaper also did that in 1932 to discredit the Los Angeles Olympics:

Negroes have no business at the Olympics. Today we witness that free white men have to compete with the unfree Negro. This is a debas*m*nt of the Olympic idea beyond comparison. The next Olympics will be held in Berlin in 1936. We hope that the responsible men know what will be their duty. The blacks have to be expelled. We demand it.

When nazi-ideas became official German policy in 1933:

Bruno Malitz published an official brochure of the Nazi Party in which he claimed:

Sports and physical education are creating physical and spiritual values. The Jew is laying his hands at all things which create values, as he is destructive. So he tried to gain control of German sports for its decrement. The Jewish teaching destroys the vigour of the people. The Jewish sports leaders and those infected by the Jew, the pacifists, the Pan-Europeans have no more place in German sports. They are worse than cholera, tuberculosis, and syphilisyas these destroy only some Germans[;] the Jews, however, destroy Germany herself. And on the other side of the Olympic committee and other member states a desire to ensure the olympic spirit was held up and that Germany should not only be not allowed to dictate other nation's teams ethnic make-up but even more so that Germany would have to ensure that Jews would be allowed into the German team.

But leading members of the IOC were sympathisers of racism, fascism, even Hitler personally and couldn't care less about esprit.

Notably, French IOC president Henri de Baillet-Latour asw the need to address all this at the 32 session of the IOC 5. – 7. June 1933 and demanded a written guarantee for such provisions from Germany. But while that may sound noble, he also argued the universal fallacy of the possibility of being apolitical:

In June 1939, the IOC voted unanimously in favour of Germany organising the 1940 Winter Games, replacing Japan that had returned the right to organise the 1940 Games. De Baillet-Latour argued that the decision in favour of Nazi Germany, which had occupied the Czech rump state three months before, showed the IOC's independence of political influences.

It is argued that the Vienna meeting would have been the most reasonable chance to change the place, if ever they really cared. Or had a viable alternative at hand. Only that these were Rome in fascist Italy, Tokyo in Japan, Barcelona in troubled and soon equally fascist Spain. And gain a pro-German Lewald led the meeting. (D. L., Hulme: "The political Olympics: Moscow, Afghanistan, and the 1980 U.S. boycott", Praeger: New York, 1990.)

The IOC arranged a shrewd deal with its German members who were equally worried about the Nazis themselves and attempted to safeguard their own position: the IOC demanded that its members remain in charge of the organisation and that all Olympic rules would be strictly adhered to. When this was achieved, the American IOC-member General, Charles H. Sherrill, a former US Ambassador, demanded additionally that German Jews should not be excluded from the German team. This interference into the internal affairs of another country was unprecedented in IOC history. At a time of racial segregation, when competition between white and non-white athletes could not take place in the American South, when major US sports were still segregated, nobody had demanded that African-American athletes be given a fair chance to qualify for the 1904 or the 1932 Olympics in the United States. However, in this case, it was essential that Lewald secured such a statement on behalf of the German organisers to placate international critics.

In these respects, it seemed, all parties involved were contented; the American press celebrated its victory. The American IOC members were happy as they arrived home as victors in the battle, as they saw it, against Nazism. The German IOC members were pleased as they had defended ‘their’ Games; and, the Nazis were happy as they could host the Games in Germany. The declaration that had won the day for the Nazi Olympics was not even published by the state-controlled German press.

But the Republican American Charles H. Sherrill was keen to let the Games take place. He is remembered for

Support for dictators
Shortly after retiring from public office Sherrill proclaimed his admiration for Europe's strong men and predicted the end of parliamentary form of government, which he dubbed "inept" and referred to as "so-called democracy." In a long letter to the editors of The New York Times, published on June 4, 1933, he singled out Benito Mussolini, the fascist dictator of Italy, for praise and spoke of the "amazing betterment" of life accomplished by his régime. He wrote of Adolf Hitler, the new leader of Germany, "Whether one admires [him] or not, at least he is a leader who leads." Soon enough, he wrote, "people the world over... will follow courageous leaders."

1936 Olympics
In 1935, during the preparations for the 1936 Olympic Games, Sherrill met twice with Hitler. A modern historian wrote that Sherrill was "mesmerized by the force of Hitler's personality and charisma." In his one-hour talk with Hitler, Sherrill insisted for at least one token Jew to be included in the German team for the Olympic Winter and another for the Olympic Summer Games. Hitler refused and when he was threatened by Sherrill with an American boycott, promised purely German Olympic Games. Sherrill sent the information to the IOC president, Henri de Baillet-Latour, who did not insist on Jewish participation on the German teams. After the Nuremberg Racial Laws, only Half-Jews, with no more than two of the four grandparents being racially Jewish, were still permitted to represent Germany. With Theodor Lewald as President of the Organizing Committee for the Summer Games, Rudi Ball (hockey, Winter Games) and Helene Mayer (Fencing, Summer Games), three Half Jews calmed world public opinion.

Equally, Avery Brundage the American president of America's NOC AOC and the AAU was so keen on US participation that he even manipulated a vote he was sure to loose over boycotting the games or not.

It seems that as the games approached the public debate in the US increased and opinions published appeared to roughly equal.

The main focus was on the US, as the American team had always placed first in the Olympics with the exception of 1912, and was considered neutral. In this respect, American opinion and participation mattered. American athletes had participated individually in German sports events during the Nazi years; however, the decision was not to be taken by the athletes but, rather, by the AAU and the AOC. Judge Jeremiah Mahoney, AAU President, was against participation, while Avery Brundage, AOC President, was in favour. […]
Both sides published pamphlets, staged rallies, gave radio and newspaper interviews, and lobbied political representatives to take sides. Most of the public debate issues were outside the actual field of sports and dealt more with the question of having the most important international gathering, symbolising the search of the world’s youth for peace, understanding, and equality, under he Nazi symbol of the Swastika, itself representing an entirely different set of values.

The pro- and anti-boycott sides clashed, nevertheless, in a decisive show-down in New York in December 1935. But it was too late to ask for a transferral of the Games; the only question that remained was whether the American teams would go to Germany or not. The discussion followed along the lines of the aforementioned arguments.

At this time, Avery Brundage used two political tricks, which underlined his skill as a master strategist. Normally, the certification of an athlete for participation in the Olympics required three signatures – that of the athlete, that of the sports federation (here, the AAU), and that of the National Olympic Committee (here, the AOC). Brundage had secretly secured the agreement of Baillet-Latour that, under these extraordinary conditions, the signature of the AAU would not be necessary. Brundage described this to his friend and IOC member, Sigfrid Edström, as the "death knell for the AAU". By virtue of this secret agreement, a vote against participation and certification of the athletes by the AAU would have meant a reduction of its influence.

Brundage’s second trick was performed directly at the AAU convention. When he realised that he might still lose the final vote, he sought to stretch out the discussion through the night. By morning, he had secured several more eligible voters by telegram. With their help, he gained a final 58 1/4 vs. 55 3/4 victory. According to voting procedure, the AAU districts each had three votes (54 1/2 vs. 41 1/2 for non-participation), the past and present Presidents two votes (1 3/4 vs. 1/4 for going), and the associated sports bodies, many of which were brought in by Brundage over night – in a display of support never seen before – cast their ballot (one vote each) 15 vs. 1 in support of attending in Berlin. The next day, the AOC voted unanimously for participating in the Berlin Olympics.

Brundage also ensured that the American team was "presentable" – by excluding Jewish athletes (Javier Cáceres, Holger Gertz: "Glickmans Trikot. Die Amerikaner kuschten 1936 vor den Nazis und ließen ihre jüdischen Athleten nicht antreten", in: Süddeutsche Zeitung, 1. August 2015, p3.) Also compare Arnd Krüger: “'Fair Play for American Athletes' A Study in Anti-Semitism", Canadian Journal of History of Sport and Physical Education, 9(1), 43–57, 1978. doi:10.1123/cjhspe.9.1.43 

He was perhaps the loudest voice – after floating the idea of cancelling the games in 1933 himself – dismissing any idea of cancelation as the games approached.

In 1933

Brundage told a newspaper reporter:

My personal, but unofficial, opinion is that the Games will not be held in any country where there will be interference with the fundamental Olympic theory of equality of all races. The Olympic protocol proves there shall be no restriction of competition because of class, colour, or creed.

Otherwise he felt that the Games might be transferred to one of the other bidding cities - Tokyo, Barcelona, Rome perhaps – or they could be cancelled altogether, as they were in 1916, ironically the only other time that Berlin had been scheduled to stage them. And, of course, if they went ahead and were boycotted by enough countries, then Germany would lose out anyway, since that would be a massive propaganda own-goal. In the end, though, said Brundage, he fully expected the issue to be resolved when the IOC convened in Vienna in June. A day after the New York Times headline suggesting that the Berlin Games might be cancelled, the newspaper ran an interview with Lewald's Olympic partner, Carl Diem, who professed shock that the Games might be in jeopardy, adding that there was no discrimination in German sport.
_{–– Anton Rippon: "Hitler's Olympics. The Story of the 1936 Nazi Games", Pen & Sword Military: Barnsley, 2006.}

Even as late as the Winter-Games in Garmisch-Partenkirchen 1936:

The IOC took a middle path. When Baillet-Latour saw anti-Semitic signage peppering the German landscape he complained vehemently to Hitler, threatening to cancel the Games. The Führer relented, ordering the signs’ removal. In Avery Brundage’s personal notes, he wrote, “Baillet-Latour said to Hitler ‘You keep your law, I keep my Games.’”

Opposition to the Nazis’ racist policies emerged in the United States in 1933, when the Amateur Athletic Union voted to boycott the Games unless anti-Jewish discrimination was reversed in Germany. The AAU vote did not influence the group that mattered, the American Olympic Committee, which decided to participate in the Games. The decision was made after the committee chief, Avery Brundage, made a “personal investigation” into the matter and received a pledge from the German government not to discriminate against Jewish athletes. Nevertheless, the push to boycott the Games continued from a variety of sources, including students at Columbia College, various religious groups, and the Committee on Fair Play in Sports, a liberal organization formed specifically to oppose American participation at Berlin. To be sure, the United States had its own deep-seated problems with racism, but the 1936 Olympics provided a chance to point the finger away from home. In a March 1935 Gallup poll, 43 percent of respondents favored a boycott.

Brundage’s “personal investigation” largely entailed listening to and then believing German officials. The more Brundage publicly explained his reasoning, the more flimsy it appeared. He told the New York Times: “Germany has nothing whatsoever to do with the management of the games. The Germans provide the facilities and make preliminary arrangements, but that is all.” The Olympics, he argued, was “under the sole jurisdiction” of the IOC. Plus, he added: “The fact that no Jews have been named so far to compete for Germany doesn’t necessarily mean that they have been discriminated against on that score. In forty years of Olympic history, I doubt if the number of Jewish athletes competing from all nations totaled 1 per cent of all those in the games. In fact I believe one-half of 1 per cent would be a high percentage.” Behind the scenes, he was more direct about his feelings. When Edström wrote Brundage to complain that “all the Jews in the whole world are attacking us,” Brundage responded with an accusatory screed:

The situation on this side of the Atlantic has become extremely complicated. As you no doubt know, half of the Jewish population of the United States is centered in New York City. The New York newspapers which are largely controlled by Jews, devote a very considerable percentage of their news columns to the situation in Germany. The articles are 99% anti-Nazi. As a matter of fact, this applies to the American press generally. As a result, probably 90% of the populace is anti-Nazi. The Jews have been clever enough to realize the publicity value of sport and are making every effort to involve the American Olympic Committee. Boycotts have been started by the Jews which have aroused the citizens of German extraction to reprisals. Jews with communistic and socialistic antecedents have been particularly active, and the result is that the same sort of class hatred which exists in Germany and which every sane man deplores, is being aroused in the United States.

Brundage’s biographer asserts that Brundage “continued obstinately to see a conspiracy of Jews and Communists” and that he was blinded by his anti-Semitism.

_{–– Jules Boykoff & Dave Zirin: "Power Games: A Political History of the Olympics", Verso Books: London, New York, 2016.}

Individual calls for boycott remained strong until the summer 1936, but hopeless and some naive minds hoped that the games would change things for the better in Germany.

Non-marked non-Wikipedia quotes from –– Arnd Krüger: "The Nazi Olympics of 1936", in: Kevin Young & Kevin B. Wamsley (Eds): "Global Olympics. Historical And Sociological Studies Of The Modern Games", Research in the Sociology of Sport, Volume 3, Elsevier: Amsterdam, 2005.

For some perspective:

In 1936, which is five years after the Japanese invasion of Manchuria, the IOC gave the 1940 summer olympics to Tokyo.

While there were talks of boycot, and while the international pressure certainly played a part, it was Japan giving the rights to the 1940 summer olympics back to the IOC in 1938, which is one year after the outbreak of the Second Sino-Japanese War.

What had happened in / with Germany by 1936 was positively tame by comparison. The IOC was not in a habit of cancelling olympic games for political reasons, and certainly wasn't quick about such a decision. The Nazis had been in power for a scant three years, and neither fascism, racism, or militarism were exactly uncommon at the time. What concerns were voiced by the IOC were easily mollified by the Nazi government.