score:2
In general, the added safety in licensed taxis etc comes from: i) the reduced pool of vehicles and ii) the added number of people in the loop (dispatchers, owners, inspectors) making it easier to find a rogue one. It's certainly not the case that all drivers operating off the books are innately evil, or operating with a goal other than making a profit on driving you somewhere. It's also not the case that all licensed vehicles are driven by angels.
Having said all that, for the type of situation you describe, a plausible negative situation is the driver stopping half way through the trip into town to renegotiate the price. In general the passenger has more negotiating power at each end of a journey (you're either in a place you know or at your final destination) while the driver's power is maximised in between.