What caused the shift from annexation of territory, to colonization of territory?

Part of the problem is "colony" is a floaty term. I think there are at least three basic ways a country dominates land:

• Colony: replacing the natives with your own people (the Americas, Australia, parts of Africa).
• Empire: Exploiting the native population (India, Middle East, parts of China, other parts of Africa, Philippines).
• Annexation: Making a land, and its people a, more or less, full-fledged part of your country (Texas, Wales, Puerto Rico).

This is is all long, grey, smeared continuum. For example, one could argue that Wales and Texas were both colonized and annexed.

I think when you're saying "colonization" you mean empire. That's been going on since antiquity. The Greeks, Romans, Persians... everyone did it because it made money for the homeland.

Why didn't the British they annex India? Because India was a commercial venture by the East India Company, and they were making tons of money exploiting the natives. India was thickly populated and fairly advanced technologically, so they couldn't just shove them out of the way and replace them with British people like was done to the post-apocalyptic North American natives (more on that later). Annexation means government which costs money, better to use the local governments and keep them divided. It means Indian natives might want to become British citizens, and citizens have rights and protections and laws. They're harder to exploit for money.

In short, colonization and empire is rarely a deliberate government act. It usually begins as a for-profit venture by private citizens or chartered companies and generates so much money for the home country the entire government gets sucked in.

Looking at this historically, the major difference between "annexing" and "colonizing" is how the dominating power views the locals. For annexation, they'd offer them the rights and respect as their own population. For colonization, they consider them sub-human to be moved around and wiped out as they please.

This view of the locals isn't by accident, it's a result of economic pressures to justify exploiting the region by othering the locals. If they're not human, then they must not have any rights. Of course, respect can also come from the barrel of a gun, and sometimes natives won that respect by fighting.

Let's look at a few examples to illustrate, and I'm going to have to paint with a very broad brush. Let's start with the European invasion of the Americas.

This was a colonization effort driven by economics: the New World was viewed as a land to be stripped of resources to supply the European powers. Europeans settled there to gain economic opportunity and new freedoms. The natives were just in the way. Accordingly, they were thought of as sub-human, almost animals, treated as slaves, their land stolen, and pushed off into whatever land the Europeans didn't want (until they did).

The native Americans had just undergone an apocalypse, new European diseases had wiped out most of their population. This, combined with not being as advanced in warfare, meant they could be easily subjugated.

Fast forward 400 years to the German invasion of the Soviet Union in WWII. Though it's often referred to as annexation, this was a very deliberate colonization effort. The intent was to gain land and resources for the German people to expand into, with one teensy problem: very well armed people were already living there. Even if Germany defeated them on the battlefield, they'd still be there. So they had to be wiped out. To that end, the Slavs were declared sub-human and the German state went on a war not just of conquest, but of extermination. This failed because the Slavs resisted, they hadn't been crippled by plagues, and fought back.

In contrast is the German annexation of Austria, and Czechoslovakia. These were mostly peaceful affairs, and the population was treated well... uhh, for a German-occupied country.

Somewhere in between is the European invasion of New Zealand. This is late in the colonial era, early 19th century. Slavery is being called into question. European culture was becoming enlightened to the idea that maybe all people are created equal and we can't just take people's stuff because they look different.

The Maori, the New Zealand natives, had many advantages other native populations did not have that allowed them to win the respect of the Europeans. They had decades of mostly peaceful contact with Europeans, time to learn their languages, and most importantly, trade for tools and weapons. They were also a very war-like people, ready for a fight, and the Europeans knew it.

On the European side, New Zealand is literally the middle of nowhere. Reaching it, even in the 19th century, was difficult. There wouldn't be the endless crush of European people and supplies, nor the insatiable lust for raw materials. They couldn't simply bulldoze the Maori, though they'd try. They'd have to bargain with them as equals. Military power can lead to social respect.

As a result, while the Maori were eventually overwhelmed, they were not shattered like the Native Americans and the Australian Aborigines. They retained some power, and today make up 15% of the population of New Zealand.

The answer to "why didn't Europe colonize China?" is similar. The Chinese natives were simply too many and too strong. India as well, though it's considered a colony, was more of an annexation and creation of a puppet state. Though in both cases the native populations were second-class citizens, so the distinction between "colony" and "annexation" blurs, no major effort was made to wipe them out and replace them with Europeans.

The American annexation of Texas was part annexation, part colonization. Texas, at the time, was part of Mexico and numerous native tribes also claimed the land. A mix of Spanish and native people were there, as well as an increasing number of white settlers. One of the major reasons for its annexation was to provide more land and resources for American settlers. Once annexed, it was colonized, but the natives were not pushed out and today the population remains a mix of white, native, and Spanish people.

This is one lens to look at colonization vs annexation with: power imbalance and how the locals are viewed. There are others.