Upvote:2
In almost all cases, the baggage rules of the first carrier on an itinerary are what apply. This is mostly for practical reasons since the first check-in agent is the only one to see them.
So, the fee schedule used would be that of American, not Qatar.
But, the only way to pay $500 for two bags is to pay both Oversize ($150) and Overweight ($100). So, if the agent determined both were overweight and overside, $500 sounds right. They should be able to tell you exactly what baggage fees were applied on your PNR. You should have also gotten a receipt at the counter.
Unfortunately, contesting this will be very hard, if even possible since you essentially agreed to the fees when checking in. If the bags were <62in, that was the time to measure them.
This pattern applies not only to weight restrictions, but over all fees and benefits as well. For example, if you have Frequent Flyer status on the first carrier, but not on the second, your will still get you bags checked though free.