Given Hitler's Austrian ancestry, why did he develop German nationalism rather than Austrian nationalism?

One reason why Hitler did not become an Austrian nationalist was because it was impossible at that time. When Hitler was a child and a young adult deciding his world views, there was no Austrian nation, but an Austrian Empire containing citizens of many nationalities.

So Hitler could have become an Austrian Imperialist as he was more or less taught in school to be. But since Hitler had a nationalist personality instead of an imperialistic personality he felt impelled to be nationalistic, and being German speaking German nationalism was a lot more natural and likely for him than Czech nationalism, Hungarian Nationalism, Polish nationalism, Italian nationalism, etc., etc..

Hitler never liked Austria or the Austrians. He tried enlisting in the Austrian army but they wouldn't let him, before that he applied twice for the academy of arts and against he was declined entry. That's was when he left for Germany and enlisted there in the army.

Mainly he didn't like Austria for the above reasons and because of the government.

Until the 1800s Germany was divided into many different countries. When the concept of nation states was developing, the general idea was people who spoke the same languages was the same nation. By some historical accident Austria was excluded when most of them formed united Germany, but it's people were still considerd Germans. The idea of a separate Austria nationality developed after the war.

The new Republic of Austria, the German-speaking rump of was left of much larger entity after World War I, was legally prevented from reunification with Germany. Hitler was one of those who opposed the ban, and his committed atrocities in World War II were such that unification can no longer be mentioned in polite society even by today. Yet at the time the idea was strong and not based on ideology alone: e.g. both Hitler on the (extreme) right and Victor Adler on the (moderate) left were among its supporters for different reasons. It was shades of gray that were later obliterated and buried.

Also consider this: Hitler was 29 years old when Austria came into existence as a nation state (or "nation state" to some). His parents had both died before 1918. His hometown (Braunau) is situated on the river Inn, right opposite the German (Bavarian) border and about two thirds towards Munich on an imaginary line that connects Munich and Vienna.

The Austro-Hungarian Empire was a polyglot nation of Germans (Oesterreich or "Austria" is German for Eastern Reich), Hungarians, Czechs, Slavs, etc. To Hitler and some other German Austrians, the only part of the country worth mentioning was "Austria," the German part. In his own mind, Hitler was "German" first, and only "Austrian" second. As the German Chancellor, he was glad to first annex the relatively "pure" Austria to Germany, then gobble up the non-German parts separately as "colonies" or satellite states, rather than part of "Greater Germany."

An American might imagine that if Canada broke up into numerous English and French speaking provinces, that at least some of the English-speaking provinces (and people) would rather be part of the United States than either "independent" or aligned with the French speaking part.

As an addendum to @Relaxed's answer, it's worthwhile to point out that Austria actually tried (twice) to annex Bavaria in the late 18th century. These attempts were frustrated by other European powers, chiefly Prussia who actually went to war with Austria over it, the so-called Potato War.

Curiously, the only gain that Austria made at the settlement of this war was the annexation of the town of Braunau - where Hitler was to be born 120 years later!

I don't know if Hitler was aware of this but it certainly goes to show that his "Austrian descent" was a bit shallow, Braunau being an area which was probably culturally very much within the Bavarian (thus "German") orbit.

@Relaxed is right to point out that Germany was unified under Prussian, not Austrian hegemony. Prussia’s 1866 military victory over Austria at Koeniggraetz definitively shut out the Austrians. Subsequently, the 1870 war between Prussia and France, with many of the remaining German states outside Austria joining in, led to William I of Prussian being crowned German emperor in 1871.

Austria, besides being half of the Austro-Hungarian dual monarchy, was in itself a Vielvoelkerstaat or multinational state – besides German-speaking Austrians, there were Czechs, Poles, Italians, Slovenes, Ruthenians, etc. Emperor Franz Joseph I, who had been on the throne since 1848, tried to maintain the loyalty of all his subjects regardless of ethnic or linguistic affiliation, but ethnic tensions were rife.

As Brigitte Hamann’s Hitler’s Vienna describes, the future dictator entered a politically charged atmosphere in high school in the provincial town of Linz from 1901 onward. His favorite teacher, Dr. Leopold Poetsch, was a German historian who also gave public speeches. Hamann quotes Hitler about him:

He used our budding nationalistic fanaticism as a means of educating us, frequently appealing to our sense of national honor. By this alone he was able to discipline us little ruffians more easily than would have been possible by any other means. (Hamann p. 13)

Yet Poetsch was at the same time an “Austrian patriot” who later was reportedly upset at being cited as an inspiration by Hitler in Mein Kampf. The Pan-German students were actually more radical than their teachers, who as public servants had to stay kaisertreu to some extent. Even at that time they supported an Anschluss, or having the German part of Austria join the rest of Germany. The “Heil” greeting (later so familiar!) was one of their identifying symbols, as opposed to the “Hoch” of the loyalists. The young Hitler would do things like distribute pencils with the red-gold-black German colors during class.

Hamann (whose book I recommend) sums up Hitler’s early German nationalism quite well. (The sections in italics are quotes from Mein Kampf.)

Later Hitler liked to emphasize that on account of their experiences in the multinational empire, the German-Austrians had developed a much more alert and progressive form of nationalism than had the “Reich Germans,” even early on, when they still attended school: In this way the child received political training in a period when as a rule the subject of a so-called national state knew little more of his nationality than its language. At the age of fifteen, Hitler reported, he had already realized the distinction between dynastic “patriotism” and folkish “nationalism.” At any rate, even at that early age he clearly joined the camp of the radical “folkish nationalists,” rejecting the multinational state as did the followers of Schoenerer [a Pan-German politician]. (pp.14-15)

In other words, Austrians like Hitler liked to think they were “more German than the Germans,” and growing up in a state being torn apart by ethnic tensions was the context for (and first step in) the process (so perverse and distressing to contemplate) of Hitler’s development as a national radical. “Austrian nationalism” would be an oxymoron in this context.

Hitler's ultimate motive was 'Upliftment of the Aryan Race'. For this an auxillary goal was making the 'pure' German Nationality walk tall, and be strong.

• Hitler was not a exactly a German nationalist, he was a Pan German, strifing for uniting the German race. This essentially opposed the rise of sub - nationalism among various states of German nationality. An example of this is the Beer Hall Putsch, in Munich for which he was imprisoned, during which he wrote Mein Kampf.

• Also, Hitler saw the Austrian state, as rotting down, due to mediocrity. The increasing Balkanization of the state made him loath the Austrian monarchy, which was 'corrupting the purity of German nationality'. As he writes, Germanisation of lands cannot be effected, by making a people speak the German language.

• Lastly, according to him those who considered a nation as an end in itself were mistaken.

At the time nation-states (and in particular France) consolidated themselves, the governance of the German-speaking parts of Europe was based on an older model, small principalities loosely associated in large empires. Consequently, many German thinkers developed a view of the nation as a bound based on ethnicity and, in particular, language and transcending political boundaries.

By contrast, by the 19th century the borders of the mainland part of France were very close to what they currently are but several peripheral regions had distinct cultures and languages. French thinkers therefore argued that nations should be based on will and free choice rather than some pre-existing cultural uniformity. In a way, the French language was something that came “from the top” to realize the unity of the nation rather than a given.

In this context, a major debate in German-speaking lands was how to realize German unity and what its perimeter should be. Specifically, Austria and its large multi-national empire represented a big problem and the alternative was between a “small German solution” (unifying Bavaria and the weak states in the North around Prussia as a pragmatic first step) and the “big German solution” (including as much as possible and in particular Austria).

In the second half of the 19th century, after several attempts and intermediate steps, the small German solution prevailed, not as a result of liberal revolutions but through the wars engineered by Prussia under Bismarck. But the idea of a larger German nation did not die out and continued to inspire an influential political movement.

Almost fifty years later, after World War I, Austria-Hungary crumbled and most of its German speaking parts formed a new Republic that was barred from forming a union with Germany by the victorious allies. Consequently, in the 1920s and 1930s, it would make sense for nationalists not to feel any loyalty to a state that did not even exist a few years before and did not correspond to their views of what their nation was.

Incidentally, from a cultural standpoint, there are still significant differences between Northern and Southern Germany today and some discernible nuances between various regions but Austria and Bavaria share a lot, e.g. linguistically. German dialects really form a continuum but the main dividing lines are not between Germany and Austria.