Was the penal system used to colonise Australia?

The British judiciary has a very long history of independence. The government (executive branch) wouldn't be able to set quotas for transportation, in the same way they can't set quotas for prisoners. The sentencing is entirely up to the judiciary, particularly before the 20th century advent of "sentencing guidelines".

Mostly, the punishment of transportation to the colonies was used as alternative to a long prison sentence, when the crime wasn't quite worthy of hanging. Generally, the idea was to get these people away from society. The colonies were not quite considered "properly civilized" and therefore little thought was given to the effects the arriving criminals would have there.

Margaret Atwood is half right and half wrong. Yes, the British had a strategy to colonise Australia via the penal system. No, their motivation for sending criminals to the Australian penal colony was not to force a large number of people to move to a yet-to-be established colony. Yes, the male-to-female balance became problematic. No, they did not simply lower the bar for women to be sent.

As for your ancestor being sent to Australia for stealing a button... England's laws were extremely punitive and harsh at the time. The 18th century is now known as the time of the "Bloody Code", a particularly stringent set of laws in England. Patrick Colquhoun wrote 'A Treatise on the Police of the Metropolis' in 1800; in it he lists crimes punishable by death and crimes punishable by transportation. Crimes punishable by transportation included "Petty Larcenies, or Thefts under one Shilling", which is probably what your ancestor was convicted of. Also, many criminals found guilty of a crime punishable by death had their sentence commuted to transportation by sympathetic judges or juries. This system of punishment had nothing to do with creating overseas colonies: it was a legitimate attempt to reduce crime in England, with a lot of people being made homeless and poor due to practices like Enclosure. There was no ulterior motive - just a desire to punish criminals and prevent crime, in a time of great social upheaval.

Previously, the British government had been sending some convicts to its colonies in North America. However, due to a local uprising there in 1776, that was no longer an option. Therefore, they were casting about for other possibilities.

The jails in England were over-full; the government was hiring old ships to house convicts on the Thames; they feared plague starting in the crowded prison ships and spreading to London; they feared the escape of prisoners back into society. Sending convicted criminals to some remote location far from London had a lot of benefits. Parliamentary committees were set up to investigate and assess the various options available (one option proposed was a remote location in Africa!).

However, James Cook and Joseph Banks had recently been to the east coast of New South Wales (if you have to name a new land and make it sound attractive, name it after an existing place that's lush and green, like South Wales - even if the new land isn't quite so lush and green...), a land in the southern hemisphere they'd claimed on behalf of King George III. Banks recommended a site they'd visited there: Botany Bay. It looked fertile and temperate, and the natives looked easy to intimidate.

There was also the side benefit of having an outpost near the Pacific Ocean, to prevent the French from gaining too much influence in this area (they were out trading and colonising as well).

The prison settlement at Botany Bay in New South Wales was never intended to be the core of a new British civilisation Down Under. It was mainly a way to get rid of undesirables. And there were a lot of undesirables to get rid of as a result of the "Bloody Code".

Later, after the convicts had started being sent to Australia, the Australian Governor asked for some free settlers from back home. There was hope that the marines sent out to oversee the convicts might start the process of settlement but, as Governor Phillip wrote to the Home Secretary, Lord Sydney, in October 1788:

Most of the officers have cultivated a little ground, but is merely for their own conveniency, and none more than a single acre. [...] It must, my Lord, be settlers, with the assistance of the convicts, that will put this country in a situation for supporting its inhabitants;

Free settlers were bribed with grants of land and free convict labour to use on their farms.

And, as for lowering the bar to send out more women... that was hardly necessary. With hundreds and thousands of women already sitting in prison hulks in England due to the "Bloody Code", it was merely a matter of choosing more women to send out in the next convict fleet to Australia. They didn't need to create more women convicts in England - there was already an over-supply.

But, these were after-the-fact considerations. Atwood has it backwards: Britain didn't change its laws to create convicts to settle Australia; they settled Australia because they needed a place to get rid of the convicts created by their punitive laws.

What about both? Australia was harsh, far away, hard to get away from. Perfect penal colony, if a tad big maybe.
Originally it was no doubt intended as a place to dump convicts when the prisons at home were overflowing. As to your uncle, remember the penal system at the time was rather less lax of "petty crimes" than it is today. Theft was theft, no silly "it's only a candybar, let's let him go with a stern warning never to do it again" attitude of law enforcement.
Usually though, for crimes like that you'd not be sentenced to be a transportee for a first offense.
And while the cost of sending you over would be considerable, you were put to work there. As a result in the end you'd probably turn a profit for the Crown if you survived long enough.
That work ended up building Australia (and areas of the Americas) so they became more hospitable to other colonists.
Also, many times a transportee wasn't allowed to return home after his sentence was up, he'd become a trustee and lived in growing towns and villages around the prison compounds, working a hopefully honest job. Others were allowed to return home but you'd have to pay for the trip, which often meant you'd spend years or decades as a civilian in the colonies trying to save up enough to pay for the ticket. And by the time you had enough, quite possibly you were married with children, you had your entire life there and no reason to go back to the old country.
As the colonies grew, so did the demand for cheap labour for the mines, farms, and public works.
As this was largely provided by the prison camps, adding to the population of those was a logical thing to do, which meant deporting more people (and thus introducing transportation as a penalty for ever smaller crimes, and for ever longer periods). But that led to greater demand for guards, thus more colonists.
France for a period had the same thing going in French Guyana and parts of Africa.

Eventually the civilian population would have started taking up the mining and other hard labour jobs as more and more people ended up in the colonies who lacked other skills (including former transportees of course) and instead of calling for more transportees there came to be resentment towards them taking up jobs that colonists could do as well and with no need for armed guards.
That's btw also a reason to abandon slavery, unemployment among poorly educated people who start seeing jobs they could do performed by slaves instead.