What are the limitations of a ScotRail Off-Peak Day Ticket?

Day off-peak tickets generally allow you to break your journey at any station:

You may start, break and resume, or end your journey at any intermediate station along the route of travel on Off-Peak tickets unless the ticket restriction for the journey you are making does not allow it. If you intend to start, break and resume, or end your journey at any intermediate station, please call 03457 48 49 50 to check if it is available on your specific journey.

(source)

However, to my understanding, you cannot continue your journey from Glasgow during the evening peak on an off-peak ticket.

(Edit: For details about peak hours, see the comments by mrpursuit)

I don’t see any limitation of number of travels

Maybe it’s necessary to clarify this – ‘Day ticket’ means the ticket is valid on a particular day. You may use it only once for each part of the journey. This is different from day passes or travelcards, which are valid for any number of journeys.

TLDR: Yes you can do this, you must make sure not to travel on peak trains for the whole of your journey. For a more detailed explanation of why and with instructions to determine it for other journeys see below:

The term you’re looking for here is a break of journey. The national rail enquires websites states about these:

Break of journey is allowed on the outward portion of Off-Peak tickets unless otherwise indicated by a restriction shown against the ticket’s validity code and in all cases on the return portion of Off-Peak return tickets.
You may start, break and resume, or end your journey at any intermediate station along the route of travel on Off-Peak tickets unless the ticket restriction for the journey you are making does not allow it. If you intend to start, break and resume, or end your journey at any intermediate station, please call 03457 48 49 50 to check if it is available on your specific journey.

source: http://www.nationalrail.co.uk/times_fares/ticket_types/46590.aspx

So in general you can break your journey, but lets quickly deal with the ticket restrictions section. Here is an example of a ticket with additional restrictions:
https://farm6.staticflickr.com/5598/15280434598_97601ddd83_o.jpg

This ticket has a FE restriction code (look in the lower right hand corner, by the text “See restrictions”. Now you can go to http://www.nationalrail.co.uk/YOURCODE (so: http://www.nationalrail.co.uk/FE) and read the additional restrictions, in this example there is nothing in the additional restrictions regarding a break of journey.

Finally, there is one more thing to consider. Is your new journey valid? For example, if you were to use your ticket between Edinburgh to Helensburgh to try and travel Edinburgh to London. Get out and then travel from London to Helensburgh this would not be a valid route. There are two criteria for determining if a route is valid, your route only needs to meet one of them.

1. Do direct trains between a and b stop at your intermediate station? In this case direct trains between Edinburgh and Helensburgh stop at Glasgow so you can definitely break your journey there.
2. The second option is to determine if your route is “mapped” as per the routing guide. This is quite a complicated process and can involve arguments with some members of staff, know your rights and be prepared to argue with them.

First go to http://data.atoc.org/rp_calc and enter your route, date and any restrictions (see above). Doing this will show that from a legal point of view, your journey from Edinburgh to Helensburgh is legally a journey from EDINBURGH GROUP to DUMBARTON GROUP. You then need to go to this list http://iblocks-rg-publication.s3-website-eu-west-1.amazonaws.com/yellow_pages.pdf (caution, very large (2268 pages) pdf file). And look for your route, it’s on page 574. In the permitted routes sections we see “EG+FW”. If your route only has one map (1 two letter code) then you must not leave the map, however, yours has two (EG and FW). The rules goverening these routes are on page F8 of http://iblocks-rg-publication.s3-website-eu-west-1.amazonaws.com/nrg_detail.pdf but to summerise:

If the route code has several maps, route tracing must observe the following rules:

1. Tracing must start at one of the ends of the route code.

2. All maps must be used.

3. Maps must be used in strict sequence, first to last or last to first, for a return journey.

4. Tracing may move from map to map only at a point where the two maps touch.

5. Once route tracing has left a map it cannot return to it.

6. Journeys may not double back except between stations which are members of a routeing point group for interchange or unless an easement permits it (http://www.atoc.org/clientfiles/files/easements.pdf).

You can use this website (http://data.atoc.org/routeing-maps) to display maps of the codes found in the PDF and to help determine exactly what they mean.