How would ancient Greeks have exercised?

SHORT ANSWER

Many of the exercises the Ancient Greeks did would be recognizable today, including push-ups and pull-ups. For equipment, they often used things that were part of their environment such as stones and animals, but there was also some equipment that was custom-made.

Contrary to what one might think, the Ancient Greeks had a good understanding of physiology. They were capable of devising complex training regimes and were aware of the importance of nutrition.

DETAILED ANSWER

Exercise has a long history and certainly pre-dates the ancient Greeks. On training for warfare

Ancient military training had similarities to the movements performed in nature by our cavemen brethren, but with more structure and a different end goal. Young men practiced fundamental skills such as walking and running on uneven terrains, jumping, crawling, climbing, lifting and carrying heavy things, throwing and catching, unarmed fighting, and weapons training.

On the theme of war, dancing was also used for young Spartans starting their training. They

...learnt the pyrriche, a dance while carrying weapons which trained the young man in his movements under arms.

Source: N. Sekunda, R. Hook, 'The Spartan Army' (1998)

The ancient Greeks, like those before them, made use of their environment. For example, they

weight trained with activities such as stone lifting, stone throwing, wrestling and rope climbing. Training in ancient Greece, particularly for the Spartans, was structured and extremely intense.

....They used body weight exercises such as pushups or pullups. The Ancient Greeks would use resistance in their strength training methods by using stones, logs, animals or each other to help increase their strength.

"Athlete lifting stones, probably for weight training, although his tucked-up garment suggests manual labor. Interior of red figure kyhx." Source: W. E. Sweet, 'Sport and Recreation in Ancient Greece' (1987)

The Greeks used Halteres (pictured below, sort of stone dumbbells) for sporting events and, according to Philostratos (cited in Sweet) for exercising.

Attrib: Portum at English Wikipedia [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], from Wikimedia Commons

Exercising took place both in the Gymnasium and outside, using the terrain. For example, for the latter there were

...daily physical activities believed to enhance conditioning, such as digging, horse riding, walking, hunting and fishing.

The same source also cites a number of other exercises, some of which could have been done indoors or outdoors:

Punching bags were used, as well as shadow-boxing techniques. Bends were used to strengthen the upper body. Various running exercises, including high-resistance running in sand, were employed to improve lower body fitness and aerobic performance. A variety of jumps are also described, while upper body strength was cultivated using rope climbing and other instruments.

Training for athletes was taken very seriously:

A structured training regime in ancient Greece included three stages: warm-up, training and cool-down – much in line with current advice from the American Heart Association.

Contrary to what one might think, the ancient Greeks were certainly not in the dark about physiology. For example,

Hippocrates and Galen, best known for their contributions to medicine, observed athletes while they trained, in order to understand the human body, and developed anatomical and nutritional guides to improve performance.

Also,

An enormous body of knowledge went into sports training in Classical Greece, much of which remains in use today, or is being rediscovered. Most impressive is that training was a total discipline, combining elements of biology, physiology, ergometry and sports medicine, and was fully integrated with philosophy and politics.

However, balance was considered by at least some ancient Greek writers to be essential. Note this from an abstract on texts of the Corpus Hippocraticum, Plato and Aristotle:

...health is the consequence of an adequate balance between the four major body humours and between feeding and physical exercises; instead, illness comes when this balance is broken, either for lack or, as in the case of the athletes, due to an excess....Athletes' way of life and training....neglect the development of intellectual and moral qualities.

Exercise was thus not just recommended for athletes, but for everyone. Anton Powell, in Athens and Sparta: Constructing Greek Political and Social History from 478 BC, cites Xenophon's Oikonomikos, in which he relates this to housewives; those who want to be the 'model wife' are

...advised to do work which involves some physical exercise, such as kneading dough, shaking and folding cloaks and bedclothes, and walking around to supervise the slaves. The exercise, it is said, would give her a better appetite, make her healthier and help her appearance to be more alluring to her husband than that of the slave women.