score:2
Let's consider one aspect of the fight against Britain; airpower. The Germans lost something like 1887 planes in the Battle of Britain. At the very least, most of those aircraft could have been saved by the Germans' not fighting in the Battle of Britain.
Japan's Admiral Yamamoto once valued the materials used to create the battleship Yamato as equivalent to that of 2000 bombers. By this measure, building the two (smaller) superbattleships Bismarck and Tirpitz cost the Germans something like 2500 bombers. And, of course, they were directed only against Britain.
The above numbers compare to the 2800 aircraft (the crucial arm) with which the Germans undertook Operation Barbarossa. The Germans could have undertaken Barbarossa with more than twice as many planes as they actually did. Some of these extra planes (and the accompanying fuel savings) could have been used in "strategic bombing" of Soviet factories and oilfields.
So yes, minimizing the war with Britain could have made a large (favorable) difference for Germany in the East. You can decide for yourself what the impact of that extra airpower could have been.