score:5
The VWP is a United States only program -- it is part of the rules for when and how you can enter the United States and/or be present there. Once you're outside the US, there is no way the VWP rules can force you to leave Canada; that is a purely Canadian matter.
So it is utterly certain that if you stay in Canada after the end of your original 90-day period, and later leave Canada without transiting through the US, then you cannot possibly be in trouble with the VWP.
If you need to transit through the US on your way back, things are murkier. Different sources (even different subdivisions of the US Government) give conflicting information about this, and it is often hard to trace it back to definitive legal sources. Some of it may be rumor, other parts of it may be administrative practices.
The extreme cases are pretty well settled:
You are certainly OK if you enter the US on the VWP and travel around in North America such that your last departure from the US is within the 90-day status you got on your first entry.
You are certainly not OK if you plan to build a life in the US indefinitely and expect to get a fresh 90-day tourist status periodically just by making a short trip to Canada and back.
The grey area between those extremes is where the confusion/uncertainty exists. The most trustworthy (according to my unscientific estimation) sources seem to say that what it really depends on is whether you can convince the border guards that you're not in case (2) each time you need to enter the US.
Here common sense would be that if you can show it has been several months since you left the US, then you're not merely doing visa runs, and the border guards should be likely to believe that you're only in transit. In order to make sure you can make that argument, look up your electronic I-94 history after you arrive in Canada and check that your departure from the US has been recorded. Then bring a printout of that that you can show to the border guards in case of trouble.
However, there are also sources -- or at least persistent rumors -- that claim that you will be in trouble unless you are squarely in case (1) above. You will have to decide for yourself if you think this is enough of a risk to settle of an itinerary that keeps you out of the US on your way home -- after all, what the random-people-on-the-internet will tell you here is just rumor.