Upvote:3
The details will vary based on the airline, airport, country, your citizenship etc, but most commonly, you will be returned to the country you came from. If this is not possible due to visa reasons etc, you will be sent to a country that is willing to accept you, usually meaning your country of citizenship. More details including chapter and verse from IATA regulations here.
Because the denial in this case is from the airline, not the local authorities, you will (probably) not be detained and are thus free to make your own arrangements. The airline will likely suggest using your return ticket if you have one, but visas etc permitting you may be able to enter the transit country or fly to a different destination.
That said, it would be highly unusual to be denied boarding at the gate for immigration reasons, because your documentation for your entire journey is checked before you board your first flight. Being denied at the gate would require unusual circumstances like losing your passport or your destination country belatedly flagging you as undesirable.