Upvote:6
If (and how much) the airline can charge depends on the contract between the passenger and the airline. Usually the contract will state, somewhere in the fine print, that the passenger has to pay the costs if the entry is refused and the destination country forces the the airline to transport the passenger back.
The obligation of checking entry papers is also written down in the contract. Usually this states that the airline can refuse boarding if the papers are deficient, but that the airline is not held responsible if it transports the passenger but entry is refused.