Schengen regulations consistently use “refusal” when an application has been examined and a negative decision has been reached, as shown by the form reproduced in Gayot Fow’s answer and by the Schengen Visa Code, e.g. article 32:

Refusal of a visa

1. Without prejudice to Article 25(1), a visa shall be refused:
   […]
2. A decision on refusal and the reasons on which it is based shall be notified to the applicant by means of the standard form set out in Annex VI.
3. Applicants who have been refused a visa shall have the right to appeal. […]
4. […]
5. Information on a refused visa shall be entered into the VIS in accordance with Article 12 of the VIS Regulation.

A refusal implies a specific stamp and an entry in a database. The person who applied for the visa should be provided with a standard form mentioning a specific reason (from a list) and explaining how they can appeal the decision. The application fee is not refunded.

On the other hand, there is no specific term for “denial” or “rejection” (the words do not appear in the regulation). There is an article about consular competence (article 18) and another one about admissibility (article 19).

Here are the relevant bits of article 18:

If the consulate is not competent, it shall, without delay, return the application form and any documents submitted by the applicant, reimburse the visa fee, and indicate which consulate is competent.

And that comes from article 19:

Where the competent consulate finds that the conditions referred to in paragraph 1 have not been fulfilled, the application shall be inadmissible

If a consulate isn’t competent or an application isn’t admissible (say it was lodged too far in advance), then the consulate obviously won’t issue a visa but the difference with an outright refusal is that the applicant should get their money back, there is no database entry or refusal stamp and any biometric data collected should be destroyed.

So the terminology might not be completely obvious but the difference between those outcomes is quite important for the people concerned.

About 95% of the questions on this site are about the UK or Schengen members, so let’s take those first.

The first exhibit is a Schengen refusal form…

…we can see that the term is unambiguously ‘refusal’. This is a standard form used by the 20+ Schengen members.

The next exhibit is a (redacted) UK notice…

…again we see the term is unambiguously ‘refusal’. Granted that the UK has changed their logo/letterhead since this image was made, but the form itself is the same. Part 9 of the Immigration Rules is entitled “General grounds for the refusal of entry clearance, leave to enter or variation of leave to enter or remain in the United Kingdom “

The rest of the Commonwealth follows this pattern. The next exhibit is from the Canadian Government…

Finally we come to the USA. Their usage is inconsistent, but we can see these exhibits…

‘Refusal’ is used in both cases. The USA has another form where they use ‘denial‘. Interestingly, they use ‘rejection’ in the same way this site’s wiki does…

…note that fees and paperwork are returned to the applicant.

That covers the English speaking world pretty much, I assume your enquiry was constrained to English. The only nation using the terms inconsistently is the USA.

I understand that an unqualified Google search will turn up countless references of many terms. ‘rejection’, ‘denial’, and so on. In seeking precision, it’s best to limit your search results to sovereign governments because they are the only ones who can issue visas in the first instance (note that you asked about ‘nations’).

I also understand there’s lots of images out there with a big DENIAL stamp on them. Looking closer, you’ll see that these are provenanced by immigration advisers using scare tactics. They do not issue visas and have commercial intent in their images. Such exhibits do not count in a serious discussion about terminology. Downloading a few, you can even see the Photoshop stamp in the metadata. And nobody, anywhere puts a stamp on the portrait page of a passport.

There will always be blogs and correspondence and forum questions that use the terms they are accustomed to. Of the 46 questions on this site to date for visa refusals, this answer is correct for about 43 of them.

Adding due to commentary

“Rejection” is much more elusive and mercurial term to nail down. This is because it’s usually handled via email or telephone and few people are going to use an internet resource like Stack Overflow to ask about it. The next exhibit is a heavily redacted email sent by a Home Office official (in Croydon) to their external stakeholders.

…as can be seen the author is using ‘rejection’ as though he expects his audience to be familiar with the term (reasonable, since that’s the business they are in). But more importantly, he makes a trenchant differentiation in his statement “…is time we do not spend making decisions…” I.e, rejections prevent them from making grants or refusals. He does not consider ‘rejection’ to be associated with decision-making.

Also on the same point, they published a strange and mysterious Freedom of Information Act response earlier this week (11 Feb 2015). It is entitled “The total number of rejections made on Indefinite Leave to Remain” and can be found here. I would not even attempt to understand its value or who asked for it. However, they are not talking about refusals. Refusal statistics are published quarterly by the Office of National Statistics so this is something else entirely: visa rejections. The Commonwealth makes similar publications and follows the same conventions. The Schengen convention would follow closely.