score:13
To begin with, Japan, Germany, and Italy during WWII were aggressively invading others; the Soviet Union was "on defense". After German invasion, the Soviet Union was fighting for its survival. There was no capacity for the Soviet Union to expand towards Japan during world war II.
Furthermore, the population of the Soviet Union was primarily near Europe, far from Japan. Logistically, invading an island nation on the other side of Asia would be a huge undertaking. Even mounting an invasion from Siberia to North China would present huge challenges, with very little reward.
In the 1930's, Japan had plans to invade the Soviet Union. The Japanese even started to invade from Manchuria (Khalkhin-gol) in 1939, but were crushed by the well organized Soviet Army. Despite the Molotov-Ribbentrop Pact which was signed in 1939, Soviet Union knew that Germany would be invading so they set about building their military. As Japan was busy in China, it was to their mutual advantage to sign a neutrality pact in 1941.
It is noteworthy that after Germany was defeated, the Soviet Union did declare war Japan in late 1945. However, the Japanese were more or less defeated at this point. Nonetheless, the threat of Soviet occupation of Japan likely hastened Japanese surrender to the USA.
You might also be interested in the Russo-Japanese War in 1905, where Russia and Japan fought each other in Manchuria. Both had high quality forces with very modern fighting skills.
Upvote:6
Adding to @axsv177's great answer, you need to note that the Soviet Union (SU) had a very weak naval power compared with Japan as the table in the linked Wikipedia article on Naval history of World War II shows. Japan had 16 aircraft carriers while the SU had none. And much of the Soviet fleet on the Baltic Sea was blocked in Leningrad and Kronshtadt by Finnish and German minefields during 1941–1944 and maimed by mines and air attacks according to the following Wikipedia article on Russian Navy.
After the Revolution, the Navy's restoration was slow, and only with the beginning of industrialisation in 1930 was a large shipbuilding program developed, but not accomplished before the beginning of the Great Patriotic War, the Soviet Union's portion of World War II. As a result, the Soviet Navy during World War II consisted of some old World War I-era ships, some modern pre-war built cruisers and destroyers, and a number of torpedo boats. Unfortunately, much of the Soviet fleet on the Baltic Sea was blocked in Leningrad and Kronshtadt by Finnish and German minefields during 1941–1944 and maimed by mines and air attacks, nevertheless numerous sorties by attack boats and submarines actions were conducted.
Until Japan made a strategically foolish decision to attack Hawaii which invited the US into the war, Japan was winning the war in China (Manchuria) and other South East Asian countries. The SU was very weak in the east at the time without much naval power. There was no reason for Japan to be afraid of Soviet invasion.
Japan defeated Russia in Russo-Japanese war in 1905 and the main reasons for Russia's defeat were weak Russian naval power and Russia's internal political situation. Japan had confidence in its military power in World War II, too much of which led to its defeat and surrender.