Upvote:3
Obligatory mention: An ESTA is just permission to get on a plane. It is the Visa Waiver Program that governs whether you get to enter the US and how long you get to stay.
I've also never heard of a 90 day per year rule for VWP visitors. 90 days per visit is usual. However (as you know) the clock doesn't normally reset for time spent in Canada. To not be caught by that rule you would have to provide evidence of your Canadian residency (I'm not sure if proof of Canadian citizenship would quality, since they are not necessarily the same thing). Canadian citizenship would normally exempt you from the terms of the VWP, but you may have to have a Canadian passport to qualify.
The issue is that VWP is supposed to be for 'short, occasional visits'. The CBP officer may have simply decided that having spent two and a half months in the US already you were trying to live in the US. He is entitled to refuse you entry if he believes that.
My advice: get a Canadian passport. I'm also a dual citizen living in Canada, and the Canadian passport gets you a lot of freedom to enter the US without fingerprinting and other hassle, and it gets you into Britain (and Europe) with as little hassle. The only advantage a UK passport gets you is to go in the fast line at Heathrow, which isn't nearly as useful.