Upvote:4
Caveat: I am not an immigration lawyer. Talk to one if you want to be sure
A word of warning upfront: US immigration is in general much more difficult to deal with than European, so I wouldn't assume European standards there. In Germany I can basically whip out both passports and ask them "blue or red?", which they would fine funny. In the US this would almost certainly gain a you an extended private interview with a senior immigration official.
In general it's good practice to enter and exit any country on the same passport. Sometimes governments cross check entry and exit record and using two different passports will bring up a red flag. I don't think it's illegal per se but it may lead to extended questioning when you leave or next time you enter.
Keep in mind that you do NOT have the right to enter the country (at least the US). Immigration officials can technically deny entry for pretty much any reason. They typically don't do this, but anything "unusual" increases the risk.
This being said: Depending on the border crossing, you could leave Canada with your Croatian passport and enter the US with the Serbian one.