Upvote:3
For your entry questions, as long as your nationality can enter Taiwan visa free, then yes you can leave during your layover. But you may have to pay a departure tax when you return in the morning. Departure tax would not have been included in your airfare because you were not "originating" from Taiwan, but once you pass immigration into Taiwan, your outbound flight becomes an "originating" departure thus subject to the tax. You will need to stop by the airline's check in counter to check on procedures, I believe the tax is T$300 ~ US$10.
I am not aware of anything interesting to do in the middle of the night in Taipei (others may chime in to prove me wrong ;-). I personally would seek out a hotel and get some rest. If you search on Agoda, Hotels.com, etc there are quite a few options in the Taoyuan area. Perhaps you can find one that fits your definition of reasonably priced.