Refused entry into Schengen after Ryanair checked passport. Now the airline wants me to pay a fine. What are my options?

This sort of treatment goes against the spirit of EU laws that deal with customer/passenger rights, see e.g. here. While there may not yet exist an explicit legal text one can refer to, to point out who is right or wrong here, if this goes to court it will eventually end up being decided in favor of the passenger, not just in case of the fine but likely also in case of the delays caused for the fruitless travel to Kaunas.

Edit: Just in case it’s not clear I’m Not A Lawyer, I’m not, and you should read this as opinion by a lay person and seek your own legal advice.

Dealing with the fine first, according to ICAO:

5.14 Contracting States shall not fine aircraft operators in the event that arriving and in-transit persons are found to be improperly
documented where aircraft operators can demonstrate that they have
taken adequate precautions to ensure that these persons had complied
with the documentary requirements for entry into the receiving State.

Since the OP didn’t flush his passport down the toilet or similar, it appears plain that the fine was incurred by Ryanair because they didn’t demonstrate taking adequate precautions, and the fine is theirs.

Guidance for unfair contract terms in the UK contains:

Terms under which the supplier must be ‘indemnified’ for costs which
could arise through no fault of the consumer’s are open to comparable
objections, particularly where the supplier could himself be at fault.
The word ‘indemnify’ itself is legal jargon which, if understood at
all, is liable to be taken as a threat to pass on legal and other
costs incurred without regard to reasonableness

Although the passenger does have a measure of fault in this case, my opinion is that Ryanair’s employee failed in their duty and caused their employer to incur this fine.

Regarding the cost of the return ticket:

5.10 When a person is found inadmissible and is returned to the aircraft operator for transport away from the territory of the State,
the aircraft operator shall not be precluded from recovering from such
person any transportation costs involved in his removal.

Ryanair also uses the same wording in their own Terms and Conditions:

you will be required to pay any fines levied against
us by the Government or immigration authority concerned, plus the cost
of transporting you from that country back to your point of origin or
elsewhere

Transportation costs, as mentioned by @MSalters, are not the same as the price of a ticket. I’d be inclined to dispute that in a small claims court.

The first thing I would dispute the price of the return ticket. The T&C’s, assuming they are legally binding, only allow RyanAir to recoup the cost of your return trip. The 10x price markup may the ticket price, but that does not represent the actual cost to RyanAir.

As has been vigorously argued in the comments, just because it is in their T&C doesn’t mean it is legal for them to have it there. I’m not a lawyer and it doesn’t seem as though any lawyers are weighing in on the topic so no one can speak authoritatively on the first point. The simplest way to resolve this without getting a lawyer would be to dispute the charge with your credit card once they make the charge. The UK has pretty consumer friendly credit card rules so I feel like there’s a good chance that this would be the end of it.

If you don’t prevail after this course of action, you may need to sue.

Assuming that you really are from one of the few countries from which a biometric passport is required for visa-free entry to Lithuania or the Schengen area and that the refused entry was not a mistake by the immigration officer:

The Ryanair document check before boarding is not primarily there to avoid your inconvenience, but for Ryanair to avoid all the hassle they have being required to handle your case if you are denied entry. Ryanair’s terms and conditions are similar to most airlines:

Article 13 – Administrative formalities

13.1 General

13.1.1 You are responsible for obtaining all required travel documents and visas and for complying with all laws, regulations, orders, demands and travel requirements of countries to be flown from, into or through which you transit.

13.1.2 We shall not be liable for the consequences to any Passenger resulting from his or her failure to obtain such documents or visas or to comply with such laws, regulations, orders, demands, requirements, rules or instructions.

13.2 Travel documents

Prior to travel, you must present all exit, entry, health and other documents required by law, regulation, order, demand or other requirement of the countries concerned and permit us to take and retain copies thereof. We reserve the right to refuse carriage if you have not complied with these and such other requirements as are set out in our Regulations, or your travel documents do not appear to be in order. (click here for Regulations concerning Travel Documentation).

13.3 Refusal of entry

If you are refused entry into any country due to immigration inadmissibility, you will be required to pay any fines levied against us by the Government or immigration authority concerned, plus the cost of transporting you from that country back to your point of origin or elsewhere. We will not refund any flight you are unable to use as a result of your refused entry. Any costs incurred by us on your behalf will be levied via deduction from the credit/debit card used to make the booking.

13.4 Passenger responsible for fines, detention costs, etc.

If we are required to pay or deposit any fine or penalty or incur any expenditure by reason of your failure to comply with laws, regulations, orders, demands or other travel requirements of the countries concerned, you shall reimburse us on demand, any amount paid or expenditure incurred. We may apply towards such payment or expenditure the value of any unused flights, or any of your funds in our possession e.g. gift vouchers, credit vouchers, etc. Any costs incurred by us on your behalf will be levied via deduction from the credit/debit card used to make the booking.

Ryanair’s boarding agent could and should have recognized that your travel document doesn’t entitle you to enter Lithuania, but their failure to do so does not free you from any liability.

Considering Ryanair’s goodwill attitude, I would unfortunately suppose that it is one of the airlines least likely to waive any valid claims they may have against you.

Addition: Since there are a few other heavily upvoted answers here with IMHO insubstantial interpretations of the situation (paragraph 13.3 of Ryanair’s T&C is allegedly violating ICAO recommendations – which by the way are only recommendations and not binding regulations, EU regulations and being an unfair contract term), furthermore Dean McGregor’s heavily upvoted comment on my answer claiming that United Airlines does not forward the fine to their customers, here some more thoughts:

Even if disputed in the following comments and being simply wrong, Dean McGregors’s comment has for some reason received a great number of upvotes. Contradicting the comment, United Airlines does as well in their Contract of Carriage, section 19.A forward such fines to the customer. All the larger airlines in Europe, e.g. Lufthansa, British Airways, Air France and KLM have similar regulations in their terms and conditions.

I have not found any references to disputes on this matter in UK courts, but there have been several court rulings on the subject in Germany, a country with similar consumer rights regulations as the UK (some of which even apply EU/EEA wide). A discussion on the same subject can be found on frag-einen-anwalt.de, where Lufthansa forwarded the fine for transporting an inadmissible passenger to Donetsk to the passenger. The lawyer answering the question, Holger Hopperdietzel, points out that a dispute in court can be ruled both ways, but does not sound overly optimistic:

That Lufthansa levies the fine for transporting inadmissible passenger
to you is not an isolated case. This happens now and then and I have
had several cases in my law office.

Unfortunately, there is only one court ruling in favour of the
passenger from the District Court of Aschaffenburg. During my research
so far, I have not found any other court ruling in favour of the
passenger.

The question if Lufthansa can pass the immigration fine directly to
the passenger is until now not judged upon in a higher court.
Therefore, both results can be possible, it all depends on the
argumentation.

So unless German courts regularly blatantly ignore ICAO documents, EU regulations, customer’s rights and companies’ unfair terms and conditions, the situation is not nearly as obvious as some of the other answers pretend to impose.

Ryanair (as well as other airlines) have no responsibility to check your full documentation and admissibilty. Their concern is mainly your identity.

Most airlines do more thorough checks, but you cannot rely on that. They do not know all the rules.

There is a chance, so you should try for sure to get reimbursed.