Buying a return train ticket to Southampton from London

Do I have to book the return (and even the forth ticket) in advance?

No. In Great Britain, there are two main types of ticket — Advance tickets, which must be booked in advance, are always single tickets, and tie you to one specific train; and "walk-up" tickets (mostly (Super) Off-Peak and Anytime Singles and Returns), which are flexible and allow you to travel on any train (possibly subject to restrictions on travelling in the rush hour), and can be bought at a station right before travel. If you can fix both your outward and return trips you might find it cheaper to buy two Advance tickets, which must be booked in advance, but if not, a walk-up single is almost the same price as a walk-up Return (out and back) ticket, so it makes sense just to buy the flexible walk-up Return (out and back) tickets. In that case, you can just buy them at Waterloo station (or any other National Rail station) on the day of travel; there's no cost saving to buying walk-up tickets in advance.

Note that South Western Railway don't do seat reservations at all, so there's no advantage whatsoever to buying a walk-up ticket ahead of time.

How much will a train ticket cost if I buy it in the station (for adults and children)?

Without a railcard

Anytime Day Return (no time restrictions): £47.20 per adult or £23.60 per child

Off-Peak Day Return (valid at any time at weekends, or any time after 09:00 leaving Waterloo on a weekday with no relevant restriction coming back): £44.10 per adult or £22.05 per child

Weekend Super Off-Peak Day Return (valid on weekends only, and only leaving Waterloo 11:00 or later, with no relevant restrictions coming back): £38.40 per adult or £19.20 per child

With a railcard

You can buy a Family & Friends Railcard, which is only valid when travelling with children, for £30. This gives you 1/3 off the adult fares listed above for up to four adults, and 60% off the children's fares for up to four children. I would be surprised if this doesn't work out as worth the cost of the one-year railcard just for this one trip. You can buy a railcard at a station with a staffed ticket office (for instance, London Waterloo), or online through the Family & Friends Railcard website. Once you've got a railcard you can buy railcard discounted tickets from ticket offices and ticket vending machines.

Note that this railcard is not valid for travel in the morning peak on a weekday (specifically it's not valid in the morning on journeys within the Network Area until the Off-peak Day Return is valid), so if you're travelling on a weekday you would need to leave after 09:00 if you're taking this option.

Other routes

This answer so far has spoken about the fast route to Southampton from London Waterloo on South Western Railway trains. One more thing worth mentioning is that there's another route to Southampton Central, on a Southern train out of London Victoria, which is much cheaper — but much, much slower (two and a half hours from Victoria vs one and a quarter from Waterloo). You probably don't want to go this way (if you cared that much about saving money I'd imagine you'd be getting a coach anyway), but I thought I'd mention it anyway. The tickets valid on both faster and slower routes are routed "ANY PERMITTED" (this is what you'll see when buying them from a ticket machine); the ones valid on the slower route only are routed "THREE BRIDGES".

How far ahead do I need to buy the ticket?

This is answered above — basically, you don't.

Can I book it when I see that we are finished and are on our way back?

If you're talking about Advance fares — no. It varies between train operators, but with South Western Railway who operate the faster trains between London and Southampton, you can only buy Advances up to 23:59 on the day before travel (and yes, apparently this applies to mobile eTickets too — see the terms and conditions on that page).