In the Soviet Union, how was payment enforced for public transportation?

Soviet Union existed for long time and all sorts of payment systems were tried at one time or another.

Admission to a metro (subway) had little difference from other countries: you put a coin or a token into a turnstile and it opened. In buses, trolleys and trams in the old times (I would say until 1970s), there was a special person called conductor. You payed her/him with cash and obtained a ticket. There were also controllers who checked that everyone had a ticket and fined those who had none.

Gradually the labor cost increased and they eliminated conductors: you had to buy a ticket from some kind of machine. In some buses/trolleys/trams you could buy tickets from the driver. Or you could buy a ticket outside (in shops, in special booths) and then puncture it with a special device in the bus. A controller would control selectively. By riding without a ticket you risked a fine.

I lived in a small town in late 1980-s, we only had buses (e. g. no subway, tram, etc.) Our buses didn't have the devices mentioned in Anixx's answer - instead, a bus would have several hole-punchers welded to the walls - typically, three hole-punchers, if I remember correctly - one in the front part of the bus, one in the middle, one in the back. To ride a bus, you were supposed to buy a ticket in a Soyuzpechat (literally - "Union press") booth, which primarily sold newspapers, pens, pencils etc., it was not directly related to the transportation system. One ticket was 5 kopecks at the time, they were sold in little "booklets" of 10 tickets. When you entered a bus, you would tear one ticket off the booklet and punch holes in it with the nearby hole-puncher. One ticket was valid for one bus ride, distance didn't matter.

There were occasional controllers, typically a pair of middle-aged, bad-tempered women, who would board a bus at a stop, ride it for a while, checking tickets, then leave at another bus stop and wait there for another bus. If they catch you without a ticket properly punched (e. g. with the hole pattern specific to that bus), they would fine you for 1 to 3 rubles.

There wasn't much "honor" at the time, riding without a ticket was fairly common (such passengers were colloquially called "hares"), no other passengers would typically express any contempt of such action. Some would keep an intact ticket, only punch it if they see controllers coming in. I saw an old guy at a bus stop once, he had a notebook with bus license plate numbers on each page, and tickets punched with that buses' hole-punchers between the pages. Since the number of buses in our town was relatively small, and the punchers were welded in places, he could simply find an "appropriate" ticket and "reuse".

For suburban trains, you wold buy a ticket at the station, the fare depended on the destination (no flat rate), and you didn't have to punch it, otherwise it was the same with buses, e. g. occasional controllers and plentiful "hares". With long distance trains, however, each car had a designated conductor who would check your ticket and wouldn't let you in if you didn't have one.

Well, as I have said in the related question, these meager kopecks were supposed to be (and, most of the time, were) enough to run the transportation networks. I also told why it was considered wrong to run services solely on government subsidy: this was tried during the War Communism but turned out to be extremely unsatisfactory. The idea was that, at some time in the future, the country would build Communism, money would be abandoned and all the plentiful resources of the economy would be distributed in a just and clever manner by very smart guys and gals - members of the society. But meanwhile, it made perfect sense to collect fares.

Tram networks that existed before the Revolution employed conductors to collect fares. Every tramcar had two landing platforms, in the front and in the rear. Passengers were supposed to board from the rear, buy a ticket from the conductor who was stationed near the door, proceed to the middle of the car and alight from the front. In multiple-car trains, every car had its own conductor. Rear car conductors would ring a bell to front car conductors; the first car conductor (chef du train) would then ring the bell to the driver, ordering him to depart. Ticket inspectors would check passengers' tickets and, if someone was traveling without ticket, fine that passenger and the conductor. Sounds easy, but not so easy in practice, taking in account that fares depended on distance (tickets of different prices used color coding; punched notches marked ticket validity limits) and trams were often overcrowded. In Moscow, to become a conductor, one had to pass the Knowledge of the City exam: a conductor had to be able to advise a route to any street, hospital, factory, church, market etc.

At the end of War Communism, this system was reproduced without alteration, first on trams and later on buses and trolley-buses (though buses often tended to use flat fare depending on route length). In metro, there were manned turnstiles. Initially, metro tickets were distinguished by direction, although the fare was flat. Later, this distinction was removed.

Curiously, this modus operandi affected the design of vehicles. In 1946, ZiS (Stalin automobile factory in Moscow) started to produce ZiS-154 buses, inspired by GM designs. Like in the US, they had doors in front and in the middle. However, one had to board through the middle doors and exit through front doors. This was extremely inconvenient and confusing (on the top of technical troubles), so in 1949, ZiS-155 was introduced with doors placed more suitably for the fare collection method used. Since 1959, the Soviet Union imported, in large numbers, the iconic Tatra trams and Skoda trolley-buses from Czechoslovakia. Originally, these trams and trolley-buses had three doors, but the SU ordered special variants with two doors. Only in 1970s the SU started to produce and import three-door designs. A notable exception was Leningrad (St. Petersburg). It usually relied on its own production of trams, and its tram company started using three-door designs in 1933 and used them ever since.

In 1960s this method became unsatisfactory and was reformed. In urban traffic, flat fares were introduced almost everywhere to simplify the rules; conductors were gradually replaced by the honor-based system described in Anixx's answer. There were a few farebox designs; some of them relied on a gravitational mechanism (utilizing the fact that all copper coins weighed 1 gram per kopeck) to dispense tickets but were found to be too prone to problems. Pass holders and those eligible for free rides were supposed to show their passes or documents to fellow passengers.

Suburban services continued to be serviced by conductors.

Emptying farebox receptacles was a fairly labor-intensive task. To mitigate this, in 1970s, transport companies started to sell packs of prepaid 'abonnement' tickets. Initially it was supposed that abonnement tickets should be exchanged for 'real' tickets by dropping them into fareboxes instead of money. Later, hole-punching devices started to be used to cancel them.

The honor-based system worked quite well. The older generation who went through the war, were both very conscientious and authoritative, so nearly nobody dared to cheat the system in their presence. If there was too much crowd, it was absolutely natural to pass your money through a chain of fellow riders to the farebox and receive a ticket back.

By 1980, however, social morals started to deteriorate, so fare evasion started to become more and more widespread.

Also, in 1960s, the metros saw introduction of automatic fare barriers. They operated on 5-kopeck coins; every station had a plenty of coin exchange automata for 10, 15 and 20-kopeck coins. Other monies could be exchanged at ticket counters; these also sold different kinds of passes. There was a single manned turnstile for pass holders and those eligible for free rides. If paying at fare barrier, a passenger would get no proof of payment.