How can I avoid being instructed to open the baggage at customs?

Customs officers generally have the legal authority (the details will be different depending on the country) to search all articles, and typically people, entering or leaving their country. As a matter of efficiency, it makes the most sense for them to focus their efforts on passengers they believe may be carrying prohibited items. Every country's customs service works a little differently, and will have different priorities for what they're looking for and how fast they need to move travelers through the line, but here's a non-exhaustive list of the kinds of criteria they may employ:

• Random searches. This may be part of an organized agency effort to statistically determine compliance rates by following a rigorous randomization procedure (e.g. every 50th person to walk past this point receives this type of search, allowing officers to estimate how many unsearched people are carrying prohibited items and going undetected) or a more ad-hoc effort.
• Profiling. Customs officers are looking for people who "look suspicious" based on their training and experience as to passengers who most often bring prohibited items. The exact nature of the profiling, especially around race and ethnicity, may depend on national laws and agency policies, and the particular biases and practices of individual officers. Some forms of discrimination may sometimes be legal at the border even where national laws would ordinarily prohibit it in other contexts.
• Your appearance and demeanor. Do you appear nervous or trying to hide something? Are you sweating? (I've even been specifically asked why I was sweating by an immigration officer once.) How are you dressed? What kind of shoes are you wearing?
• Where you're coming from and how long you were there
• Your luggage: type, quantity, weight, cost, appearance, etc...
• Information in government databases about you, including a past history with Customs or law enforcement, intelligence, a tip-off from someone that you were acting suspiciously, your travel history, etc... Some governments receive the entire PNR for your travel, and may apply secret risk assessment algorithms to select passengers of interest for a search.
• Trained dogs reacting to your person or bags
• Your answers to any questions, either orally or on a declaration card.
• Some countries may examine or X-ray bags before they even arrive at the baggage claim, identify ones of interest, and have officers watch to see who picks them up before performing a search.
• While a bag search is often a customs matter, officers may look at bags as part of the immigration process as well (or immigration and customs may be combined functions), especially if they are unsure whether to allow you in the country. They may be looking for anything that appears inconsistent with your stated reasons for travel, say bringing work tools and equipment on a vacation or copies of your resume and diplomas when you're visiting on a tourist visa.

The US CBP also has a not terribly helpful article: Reasons CBP Officers search passengers.

These factors can work together. A nervous-looking young man traveling alone who visited Colombia for just 24 hours, said he was there on vacation, and is carrying five large brand-new suitcases is going to raise a whole host of red flags.

There isn't a lot you can really do to avoid such a search, and I'd caution that most efforts to artificially change your demeanor will probably wind up making you appear more suspicious, as "nervous person trying way too hard not to look nervous" is not a good look. It is possible that you triggered some suspicion in an officer's eye; it is possible that the search really was truly random; and it is possible that they were singling out men for some reason or no reason at all. Mainly, you can avoid being searched by looking and acting like a typical traveler, and accept that sometimes your bags may be searched for no particular reason.

You may consider watching some of the TV programs that have been produced about Immigration and Customs officers (Homeland Security USA, Border Security: Australia's Front Line, UK Border Force, etc...) for a sanitized, government-friendly view of what these officers are looking for and how their jobs generally work.

All that said, most customs authorities do have a complaints procedure that you may use if you believe you have been treated unfairly or that illegal discrimination was used against you.