score:2
In simplest terms, once the boarding is completed and the boarding door closed, a number of pre-flight procedures are started based on the number of people on board. Performance calculations based on weight and balance being an obvious one. Additionally, the final manifests must be transmitted to various Authorities.
Changing the number of people on board requires these procedures to be re-done and that takes time causing everyone, including the other passengers, to wait.
It is also quite possible that the seats assigned to the late arriving passengers have been given to others so there would be no where for them to sit anyway.
For clarity, the staff can return an aircraft to the gate to board additional passengers it's just far too disruptive to do so. They will however return to the gate to deplane a passenger who does not want to fly as that is viewed as a risk on several levels.