At the beginning of the 20th century, how did Austria-Hungary and Russia's foreign policies conflict in the Balkans?

At a fundamental level, both Russia and Austria-Hungary were imperialist, multi-ethnic states too close to each other. Any specific events, like the assassination of Archduke Franz Ferdinand, have to be seen under the lens of this rivalry.

A century before the time you asked about, Austria stood with one leg in the remnants of the Holy Roman Empire, and with the other leg in the Balkans. Prussia and Austria fought for control of Germany and Prussia won. A few decades later Germany and Austria became allied, yet the way to dominate the smaller German states was blocked.

Czarist Russia had been on a steady growth from Muscovy towards the east, south, west, and north. Their expansion into Siberia is well known, yet they also went into Finland, Poland, and Ukraine, to name a just few. Russia saw itself as the natural leader and protector of the Slavic peoples in the area, but it was also ready to conquer non-Slavic peoples. Austria-Hungary did include some Slavic peoples.

The causes of the First World War are quite complicated, and defy a simple explanation like "nationalism," which in itself is not a sole cause nor sole symptom to any war. It's best to refer back to a contemporary source.

Liddell Hart put it best:

The fundamental causes of the conflict can be epitomized in three words - fear, hunger pride.

Peeling the onion back a layer, he has this summary:

Fifty years were spent in the process of making Europe explosive. Five days were enough to detonate it. [..] On one side we should have to trace the influence of Prussia on the creation of the Reich, the political conceptions of Bismarck, the philosophical tendencies in Germany, and the economic situation - a medley of factors which transmuted Germany's natural desire for commercial outlets, unhappily difficult to obtain, into a vision of world power. We should have to analyze that heterogeneous relic of the Middle Ages known as Austria-Hungary, appreciate her complex racial problems, the artificiality of her governing institutions, the superficial ambitions which overlay a haunting fear of internal disruption and frantically sought to postpone the inevitable end.

On the other side we should have to examine the strange mixture of ambition and idealism which swayed Russia's policy, and the fear it generated beyond her frontiers, especially among her German neighbors, perhaps the deadliest of all the ingredients in the final detonation. We should have to understand the constant alarms of fresh aggression which France had suffered since 1870, study the regrowth of confidence which fortified her to resist further threats, and bear in mind the wounds left in her side by Germany's surgical excision of Alsace-Lorraine. Finally, we should have to trace Britain's gradual movement from a policy of isolation into membership of the European system and her slow awakening to the reality of German feeling towards her.

To look specifically at Russia and Austria-Hungary, you have to look at the Balkans. Turkey was weak and being forced out of Europe. Bulgaria, Serbia and Greece went after Turkey to peel off Macedonia. Austria's "meter of concern" rose depending on how much Serbia benefited from the spoils, because the inter-play between Serbia and Austria-Hungary's ethnic situation was complicated. Serbia could become a rallying cry for the non-trivial slavic elements within the Empire, and Vienna felt that settling the issue in a foreign war with Serbia was better than an internal revolt. And so Austria-Hungary began to eye a war as a way of opening a social safety valve.

Russia was also thinking in these terms. Internal unrest had been quite serious in the past years and was building up again. The "golden ring" would be its dream of capturing the Dardanelles from Turkey. Her Balkan clients had faltered in delivering that dream (or at least showed they could never deliver such a lofty prize), and by this point Germany was building up Turkey's army, and so many Russian ministers were inclined to believe that a general European war would knock back Turkey (and Germany) enough to achieve that objective, while also staving off an internal revolt.

Indeed most of Europe was thinking in terms of war by 1914. The pro-war parties in Germany were becoming intense (despite causing the pro-peace parties to coalesce around the Socialists). Poincare in France declared that France would not seek war, but was not afraid of it either. Hart describes a sense of fatalism descending upon the continent at that time. He describes the situation as gunpowder having been strewn everywhere, with many inclined to light a match to meet their objectives.

Liddell Hart, B.H. A History of the First World War. Pan Books, 1970.