Upvote:4
I don't think such a study has ever been conducted; in general, "what if" studies are not done for "low impact" scenarios, and "Khmer Rouge in power" is certainly a low impact event for those with spare resources to undertake abstract research (i.e., the "rich countries").
However, I agree with the commenters that a typical failed state (starvation, population decline/collapse) would have persisted until either a coup by an establishment insider or a foreign invasion (as actually happened) upset the balance. A popular uprising is unlikely because of the lack of leadership, potential leaders being the first to be eliminated by a terrorist regime.