How did this currency exchange work?

Transaction was written as the opposite of what happened

This transaction is written up as the opposite of what you describe. It’s written as them selling 50,000 yen to you for $537.93 plus the $9.95 "service charge"¹, for a total of $547.88, with you tendering $547.88.

This was caused by an error in how the teller processed the transaction. The teller will have a significant discrepancy when they end their shift and attempt to reconcile the amount of cash they actually have with what is expected.² This was probably caused by the inexperience of the teller and/or poor training. Any reasonably experienced teller would have known that the big "Total Due:" number was what they should be collecting from the customer, not what they were to pay out to the customer.

What you should have gotten: $376.83

Based on the numbers you provided: You Googled that your 50,000 yen was worth about $450, or 111.1111 yen per $1.³ Assuming they use that number as the actual market rate and have the same difference (18.1624 yen/USD) between it and both their buy and sell rates, they would have sold you USD at 129.2735 yen per $1. This would have resulted in a base of $386.78. They would have then charged you a "service charge"¹ of $9.95 for a net to you of $376.83.

Thus, due to the teller’s error, you received $171.05 more than they would have normally paid you.⁴

Shop around

Airport currency exchange booths are notorious for providing poor rates for currency exchange. Just like many/most things in an airport, the deal you are getting is not nearly as good as you might get by shopping around outside of the airport. There are a good number of other options. What’s available near where you happen to be is something that you would need to search for. Sometimes, the convenience of exchanging currency in the airport is something which is worth the price. However, as you did, you should know approximately what the real value is of the money you are converting and judge for yourself if the convenience is worth the cost.

^{1. I have a hard time not putting their "service charge" in quotes when they are also charging more than a 30% spread (the difference between their sell and buy prices, not each individually to the current market rate). A currency exchange spread is generally used to safeguard against short-term fluctuations in the market, not long term (i.e. how long it takes them to re-sell the currency, not hold it for a year). Usually, such a large spread would be the service charge. Using such a large spread to limit their risk in holding the yen is not remotely justified by the historical level of volatility in the exchange of yen. You have to go back to February 2013 to see market rates outside of this range (November 2013 to get below 100yen/$, but it was just above 100yen/$ in mid to late 2016). Thus, one can only consider this wide of a spread as a service charge in itself (i.e. it’s primary purpose is to make them more money, not to guard against volatility).}

^{2. From this transaction alone, they will have 100,000 yen more than expected and $1,095.76 less USD than expected. Both of these are just double the actual currency transaction, as the issue is that the exact opposite took place than what was entered into the system. Note that this will be the discrepancy vs. what the system expects the cash will be, not what they would have had if the transaction had occurred correctly.}

^{3. 111.1111 yen/USD is probably a bit high, but it’s what the "about $450" from the question works out to. Exchange rates have varied between 108.79213 yen/USD and 112.15677 yen/USD over the last month, but was below 111 for almost all of that time. The current rate is 109.66 yen/USD. A Google search will provide you with places where you can get historical exchange rate data and graphs.}

^{4. Remember, these numbers are based on the assumption that the exchange’s spread is centered around (likely) the actual market rate on that day (for which you only provided an approximate number). Thus, the calculated numbers should only be considered approximate.}

Airport changers are rarely offer good value but this time they did. In order to make money they have a BUY and SELL rate. The difference between these two is called the SPREAD and the real exchange rate is in between the two values, commonly but not necessarily in the middle.

What happened is that they gave you the SELL rate which means that the transaction is setup to give you Yens in exchange for US Dollars. That rate is computed to be lower than the exchange rate, so that they end up with some profit from your transaction. Had they given you the BUY rate, you would have gotten less and it would have worked in their favor, but it didn’t since you got the wrong rate.