As others have said early WWII tanks were lightly armored and subject to armor penetration from rifles. These kinetic energy rounds have a single purpose of punching a small hole in the armor, and not much has changed today.

Consider that the round must travel at high speeds and only punches a small hole in the armor. Once inside, the velocity is slowed but energy is conserved. Some will be converted to heat, but presumably there will be two projectiles that are hot traveling at slower speeds. The second would come from the armor. These would bounce around on the inside of the armor causing wounds or death to crewmen, cutting lines, igniting fuel or ammunition, or damaging equipment.

Also the main benefit of armor vehicles is their mobility. The rifle could be used to get a mobility kill by doing any number of things. Damaging the engine cooling system, the engine block itself, the drive wheels, or damaging a track. Each have varying degrees of effectiveness. Repairing a track on a lightly armored vehicle would take a few minutes for a well trained crew. I am sure you have seen photos of WWII tanks carrying track sections. They are pretty easy to repair. However, a hole in the engine block, not so much.

Later in the war when tanks were uparmored, they still relied on vision blocks for drivers, gunners, and commanders for battlefield awareness.

Targeted anti-tank rifle fire shattering the vision blocks would render the tank ineffective, and could cause a tank to retire from the field.

Schwern's answer is brilliant, so let me just add a few more points:

• You don't necessarily need to destroy the tank outright. Disable its tracks (a significant weak point), and it becomes easy prey to anti-tank grenades. Introduce spalling, and you can kill or wound the crew. Destroy the periscope, and the crew can't see. Hit the sight ports, and you most likely kill the driver. Every tank has a lot of weak points, and it's a lot easier to hit those from short distance and with a steady hand from good cover.
• Penetration varies wildly with distance. A projectile that can't penetrate a tank at 1000 meters can be deadly at 100 meters. Angle is also very important once sloped armour began to be used extensively - and infantry with anti-tank rifles have lots of opportunities to optimize the incident angle, unlike artillery and tanks.
• High-explosive ammunition wasn't used much against armoured objects - so your assumption that an armour-piercing bullet can do no substantial damage is obviously false. Tanks usually shot each other with armour-piercing rounds. HE was more useful against infantry, artillery and buildings, and other little-protected targets.
• Don't think those rifles were low-powered. With few exceptions, those are recoil-less rifles, so they are not in the same power range as typical hand guns. Actually, recoil-less rifles were always the exception; most anti-tank rifles in WWI and WWII were "just" rifles. They were still powerful enough against early tanks (the first two or three "series" of German Panzers had very little armour - in fact, they could be penetrated by a (heavy) machine gun, at least from the sides).
• They weren't used against top-of-the-line tanks later in the war, but they were still very useful against many kinds of enemy vehicles that didn't have too much armour.

Also, just for completeness, the later Panzerfaust and similar weapons still have the same basic approach to penetrating armour - they are HE-powered, but the brunt of their effectivness comes from pretty much creating an armour-piercing projectile during the explosion; a shaped charge basically hurls a long bolt of dense, high-speed projectile against the armour at point-blank range. The rocket is simply used to deliver this to the tank at range (though it took a long time for these to get reliable, and most of the time they were used at even lower range than the anti-tank rifles); the anti-tank charge itself was pretty similar to the one used in anti-tank grenades long before.

Schwern has a very complete answer in regards to tanks. I want to expand on that answer - not only were early tanks a lot more lightly-armored than many people think, but tanks were not really the prime target of these weapons. Even in the pre-war period armies around the world were developing small anti-tank cannons that could be used by a heavy weapons platoon or anti-tank squad that could be attached at the company level to deal with tanks. ATRs were primarily intended to deal with scout cars and the new half-tracks.

The Germans and the Soviets produced thousands of these light wheeled vehicles; the British and the Americans less so. Armor was in the range of 5 to 15 mm, in most cases. While possible to penetrate with an armor-piercing rifle round, that ammo was rare. An ATR could punch through and ignite fuel or ammo, crack an engine block, or kill a crewman. These vehicles were used extensively in reconnaissance and screening roles, and scout patrols or pickets on both sides might frequently encounter one or two of these vehicles miles from their own supporting parent units. So some sort of man-portable "heavy" weapon was needed to deal with them. In fact, some of these scout cars were equipped with the same ATRs as the infantry carried early in the war.

Examples would be the Sd. Kfz. 221, 231 and 234 families (German), BA-64 (Soviet), Humber (British), and M-8 Greyhound (American).

These rifles could also come in handy for penetrating splinter shields on anti-tank-guns or metal shields to protect snipers (more of a WWI issue).

With all great answers I would like to demonstrate what weak spots were targeted. Here's picture from the 1942 Soviet field manual "Уничтожай фашистские танки из противотанкового ружья" ("Destroy fascist tanks with anti-tank rifle")

Here we see weak spots of the German T-III tank. They include engine, fuel tanks, transmission, gun and serve to disable tank as a fighting vehicle. The infantry could also shoot the periscopes (to blind the crew) and viewing slits (to kill the members of the crew) even with ordinary rifles and machine guns.

It should be stressed that the fire from anti-tank rifles had any effectiveness only at the very close range (100m and less!) and mostly from the sides and the back. All in all it was never a main tank-destroying weapon and some countries prefered to disregard them to put more resources into anti-tank artillery.

Another point is that in addition to the popular imagination of WWII tank there were a lot of lighter armoured vehicles like light tanks, tankettes and armoured cars all of which were much more vulnerable to such light weaponry.

Schwern's answer is very good and exhaustive from the technical standpoint. From my experience as sniper trained on Anti-Materiel Rifles (AMRs (Hecate II), I would like to add a few tactical aspects to that.

tl;dr: You aim to immobilize the tank so it's a good target for the AT weaponry. Your best bet is a good hit on the tracks.

First of all: The gunner does not aim to destroy the tank, he wants to immobilize it. In this situation the tank is a great target for others (e.g. the AT specialists or in WW2 the artillery/ship artillery, other tanks, air force, etc.) as it's a really big target and can't run away. So the gunner with the AMR just stops the tank, the big calibers finish it.

Your first target as an AMR gunner, like with the Hecate II, are the tracks. It's a huge and easy target constituting of many sensitive parts. If you miss one (e.g. the actual caterpillar track), chances are very high that you hit another vital part (e.g. from the suspensions, etc.) rendering the tank immobile anyway.

Tracks are furthermore a great target you can hit from almost any angle. No problem from the sides, still very good target from front and rear and still acceptable from top (however, in that case you opt for other parts). Even if you hit only the track shields, chances are still acceptable that A) the bullet penetrates anyway or B) parts of the shield break off and block/damage something behind (UPDATE: See here: https://youtu.be/9iL_6IyH9gs?t=358 )

Losing its tracks means that the tank has to be retrieved or at least repaired from the outside. So either the crew have to get out of their tank (anyway advisable in case of immobility) or more personnel from outside have to be brought into the danger zone - easy picks for snipers/artillery/machine guns either way!

Tracks are your best option when attacking a tank with an AMR.

Your next best choice is the engine. It's normally clearly visible where it is located (exhausts, cooling, etc.) and also works if the enemy's armored vehicle has wheels instead of tracks. The motor block is normally also a very sensitive target. Once the bullet or parts of the armor make it inside, it is very likely that they block something or cause serious damage.

Least preferable is any other option except if you are located high above the tank (e.g. on a building with a tank in the streets). Then it can be, depending on the situation (like angle to the tank), a good idea to go for the optics or even the crew itself.

In WW2 tanks (and nowadays still some older models) aiming for the ammunition stock was a good idea as well. Modern tanks are now mostly well protected against internal explosions (e.g. by predetermined breaking points, etc.).