How were middle ranking Nazis known to the allies before the end of the war?

I think your answer is best illustrated with examples on a case-by-case basis.

Sometimes the "middle-managers" would be known to Allied intelligence based on direct information gathering schemes. Other times some persons might be known to the individuals involved. For example, many of the events we'd call war crimes had witnesses- perpetrators, victims, and bystanders. Sometimes these individuals had the chance to speak out about their experiences. Lastly, the Germans helped in their own way as well: the Wehrmacht, the SS, and the Gestapo (like any other organizations) produced tremendous amounts of paperwork which could be examined and understood if captured by intelligence officers. They even produced paperwork that documented war crimes, because some such actions were sometimes ordered by higher authorities, and those orders leave a trail of documentation.

A good example to look into would be the investigation of the Stalag Luft III murders. Stalag Luft III was a prisoner of war camp for Allied aviators. The prisoners there conducted a series of breakout attempts, the most famous one documented in the book The Great Escape by Paul Brickhill. This event resulted in 76 escapees, of whom 73 were recaptured and 50 were subsequently murdered as a warning to other prisoners not to attempt further breakouts.

The investigation into these murders has been throughly documented. This event is somewhat unique in that a special investigative task force was tasked with solving just these murders, but the kind of investigation that ensued has common themes with others that occured during and after the war.

I have only read the short description given in The Great Escape, but according to the wiki page above there are whole books and television programs about the investigation. My recollection about the investigation (which is inevitably going to be somewhat wrong) was that:

1. The Allies first knew the murders had taken place because the remaining prisoners at Stalg Luft III were informed that 50 of their 76 comrades had been killed attempting to escape. This seemed outrageously high to the prisoners.

2. The prisoner's suspcions were further confirmed when they were delivered the cremated remains of their escaped comrades. These urns were normally supposed to have the place of cremation as well as the name of the person, but some urns did not have locations and some of the urns that did have names were inconsistent with the official story they were told.

3. The circumstances they were given- that all 50 escapees were shot while running away after being ordered to stop- made little sense. These were men who fully expected not to make it back to neutral or allied territory (one of the primary aims of the escape attempt was to tie up German units who would otherwise be contributing to the war effort). Once the gig was up, they would be expected to surrender specifically to avoid being shot and killed.

4. After the war captured documents had shown that Hitler had ordered the murder of a large number of prisoners as retaliation for the escape attempt. These orders were passed onto Himmler, who in turn passed orders onto the local Gestapo branches in the areas where the escapees were caught. This provided a paper trail that fingered many of the invidiuals involved with the murders.

5. During the investigation, the Allies found eye-witnesses who could corroborate when and where the recaptured prisoners were last seen alive, and in some cases who they were last seen with. These hints were critical in finding the right people to interrogate.

6. The big break in the case came when a captured Gestapo officer was caught in a lie during many difficult interrogations. He was pretending to be someone who he was not. Once caught, the Allies knew he must have knowledge of some of these events, and he ended up fingering other members of the Gestapo who had been involved. Armed with this knowledge the investigators could then track down these other individuals- some of whom were still in custody and others who were still in Germany or the surrounding countries- and in turn get more confessions. The Gestapo had issued many of its people with fake identities to use in the event that Germany lost the war, so a difficult part of this investigation was matching who they already had in custody with their true identities.

Databases such as the one you cite would have been useful tools for corroborating the existence of individuals and their positions within the German power structure, but there was a lot of old-fashioned detective work that went into bringing German war criminals to justice.

There was lots of spying and surveillance being done by the military, producing copious amounts of often quite mundane data that was then analyzed for its military usefulness. All it took was to make it someone's job to look through that same data for evidence of war crimes.

CROWCASS was initiated after the war was essentially won and the Allies had the resources of Germany and everyone in it at their disposal. They had hundreds of thousands of people in prisons and camps and gigantic interrogation teams. German officers and leaders were interned in special prisons and subjected to round the clock interrogation. Many German files, archives and records had been captured and were used to systematically identify Germans who had been in units associated with war crimes.

The British, in particular, have a long history of assembling intelligence files on people and organizations and throughout the war made comprehensive efforts to profile and itemize individual members of the Nazi organizations that they considered in any way significant. It was actually the British who mainly created, operated and developed CROWCASS. The director in charge of the CROWCASS effort was a British spy named Lieutenant Colonel Richard Frederick Luck.

If there were Allied spies in Berlin during the war, they would have been seeking information of more immediate value. The CROWCASS lists were not finished until 1947, so there would have been plenty of time to use German records after the war.

One source that may have contributed was the card-index at Bletchley Park. That indexed every name, place, ship, and anything else mentioned in any decoded German signal: those decodes were what was called "Ultra" intelligence. Given the name of a concentration camp, for example, it was easy to look up everyone who'd ever been associated with it in a signal. Source: Station X, by Michael Smith.