Upvote:3
I am not a lawyer and this is not legal advice, but this is fairly basic. It's legal in the sense that your booking is a contract with the proprietor of the B&B and you agreed to those terms by making the booking. Not following up is the same as not reading a contract.
You don't list the location but it's possible, though unlikely, that cancellation fee is not enforceable because of specific law applicable in the B&B's jurisdiction. In that case, they could only charge you what is allowed by law. You would have to look into the applicable laws to pursue that.