score:3
I've traveled to the UK several times in the same fashion (passport on the way in, residency card on the way out), and at no time did the immigration officer assume I've overstayed on my previous trip. So either the UK doesn't keep a tight track on how long people stay or they've simply matched me in the database by name/birthday.
Either way it's fine.