Refused to fly back to one's own country without vaccine or negative COVID test

Your question is much more generic. The same rules apply if you visited a region with yellow fiber, etc.

In general: if you are a health hazard, you cannot flight on commercial flights. Because COVID (and other diseases) were internationally recognized you can have such ban to flight. Public health has priority to individual freedom.

But so, how to reach your country?

In this case you should check other way to travel to your country, or a negative test (for other diseases there were also quarantine). For yellow fiber is also common: just travel to a zone without yellow fiber (which you are allowed to go, possibly without flying). Stay 14 days (depending on country entry requirement), and so now you can return to your home country (you proved with your "quarantine" to be safe).

For Ebola, some people organized special flights (e.g. air ambulance) with special equipment so they can return to home country (but still subject to quarantine). Usually it was just to go to a "western hospital to be cured from Ebola". But special flights.

Countries organized at beginning of the pandemic own flights to bring back citizens (just because airlines could not fly, or it was safe for them: risk to be blocked on a foreign country, and so many financial risk. If the flight is ordered by a government, risks are reduced, and the country will pay the airline in case of problems).

So, you are usually allowed back to your country (but then quarantine or other measures country specific), but it is your task to find a way (or your travel insurance). It is not a problem of commercial airlines, and you cannot pretend to fly with them. (an other reason to have a travel insurance).

The airline is compelled to follow the rules for accepting passengers for transport. It’s no skin off their rudder if you’re put into a tough situation for overstaying your visa because you lack the proper documentation.

It would be the same thing if you didn’t have a passport, you’d be denied boarding even though your own country would eventually let you in once convinced of your identity.

something would have to happen. But what?

Lots of things could happen. The Schengen country might make an exception for you since your home country wouldn’t let you back in, or they might stick you on a plane back anyway. If they do send you back, your home country also might make an exception, or they might not and you’d end up living in the airport indefinitely. You’d need to choose a specific pair of countries for this part of your question to have a chance at having a solid answer.

Is there any international general rule that would apply?

Sort of. Article 13 of the Universal Declaration of Human Rights says "Everyone has the right to leave any country, including his own, and to return to his country." So if a country refused any of its own citizens entry, for this or any other reason, it would indeed be breaking this rule and violating their human rights. Unfortunately, like most international law, this one has no teeth, so they’re almost certain to get away with breaking it.

There are 195 countries and each with it own sets of rules and preference. This makes for 37830 possible combinations here which makes a "general" answer impossible.

I suppose the host country couldn’t force me to take test or vaccine.

They probably won’t physically force you, but they could put you in quarantine or jail until you do. Some countries are nice, some are not.

Would the it deport me back to my home country,

Most likely: yes.

thus overriding the latter’s rules?

Many countries have mutual agreements for how deportation works. Transportation needs to be arranged and if that’s by air, the airline needs indemnification from fines for transporting a passenger who is not eligible to enter.

Chances are the home country will take you, stick you into quarantine (at your own expense) until you test or vaccinate. You might also get sued or fined by the government.

Is there any international general rule that would apply?

Yes. Most citizens have indeed the legal right to enter their home country. However that doesn’t mean that countries don’t violate this law and there is little recourse if they do. For example, the US wants all US citizens to have a valid passport when entering the US. If your passport is expired they can’t deny you entry, however they can fine the airline. So in essence they use the airline as the enforcer for a rule they can’t legally enforce themselves.

Is there any international general rule that would apply?

This is the only answerable part, everything else requires speculation.

Yes, there’s an international rule. It is the "International Covenant on Civil and Political Rights" (Wikipedia). Article 12 of that document states, among other things, that:

No one shall be arbitrarily deprived of the right to enter his own country.

Most countries of the world (almost all, in fact) are signatories to this document.

There’s an article in the Blog of the European Journal of International Law that discusses the exact scenario you’ve raised in light of this convention obligations. There’s a lot of legal analysis there, but it’s readable by laymen (that’s a blog, and not the actual journal).