Why was the "Cave of the Patriarchs" forbidden to Jews?

To prevent Jews from communicating with the spirits in the cave below is my guess.

The Cave of the Patriarchs (also known as the Cave of Machpelah) is Abraham's burial cave according to the Abrahamic religions. It used to be a major pilgrimage site for Jews, Christians, and Muslims. Sometime in antiquity, not clear when, the innermost cave chamber (where the bones are) was sealed off and Herod the Great built a rectangular enclosure serving as a shrine for Jewish pilgrims on top of the cave. During the Byzantine era, a basilica was built on the enclosure which the Muslims then converted into a mosque called the Ibrahimi Mosque (Ibrahim is the name for Abraham in Islam).

Inside the mosque there are two sealed grated entrances to the cave. Around these, several local legends grew and people believed that the spirits of the patriarchs and matriarchs still roamed in the cave and perhaps also could offer them their protection.

One legend recounts how the Ottoman sultan's sword fell into the cave and a local Jew had to retrieve it for him (Auerbach 2009, p. 41):

A story was told of the sultan who was peering through a small opening in the floor of the Isaac hall in Machpelah when he accidentally dropped his sword into the most sacred spot in the shrine, the passage leading directly to the burial caves. Several of the sultan’s soldiers were lowered through the opening to retrieve it, but each, in turn, emitted a piercing scream and was pulled up dead. Local Arabs suggested that a Jew, whose life was expendable, be commanded to retrieve the sultan’s sword. In terror, the Jewish community fasted and prayed for guidance. The elderly Rabbi Avraham Azulai finally volunteered. Dressed in traditional white burial garments in anticipation of his likely fate, he was lowered into the cave. There he encountered three bearded men who identified themselves as the patriarchs. Fearful of the sultan’s wrath, Rabbi Azulai asked permission to remain with them, but they insisted that he return the sultan’s sword lest the entire Jewish community be eradicated. He was assured, however, that within a week he would return to join his ancestors. Rabbi Azulai spent his final week of life teaching Torah to his students. When he died, he was buried in the Jewish cemetery near the Cave of Machpelah.

Another legend describes how Hebron's Jewish community received an "unexpected guest" for their prayer service (Auerbach 2009, p. 39):

The Jewish community remained so small that often it could barely summon a minyan (quorum) for prayer, even for the Kol Nidre service that marked the beginning of Yom Kippur, when “Jews in their talitot walked barefoot to synagogue, and the sun turned to the moon.” One year, a futile search for the necessary tenth worshipper left the elders in despair, unable to hold a service on the evening of their holiest day. Just as the sun was setting, according to a popular story, an old man appeared, white bearded, with torn and faded garments and swollen feet, burdened with a heavy sack on his shoulders. Because Hebron Jews were known to be “God-fearing and wholehearted scholars and sages, holy and pious, dispensing charity and showing hospitality with full devotion,” they welcomed him, and he prayed with them.

After the Ne’ilah service that concluded the holy day, while on their way to share the communal break fast, the visitor suddenly vanished. The grateful Jews of Hebron searched for him in vain throughout the night. When the exhausted beadle (synagogue attendant) finally closed his eyes in sleep, he suddenly saw the wayfarer standing before him, garbed in dazzling jewels, his face radiant. He identified himself as “Abraham the Hebrew, your father, whose body rests in the cave of Machpelah. I saw how grieved you were because you did not have the quorum of ten to pray, and that was why I came to you.” He assured Hebron Jews “a year of blessing and prosperity beyond all bounds.” To honor their righteous visitor, they named their new synagogue Avraham Avinu (“Our father Abraham”).

According to a local tradition, visitors could slip petitions through the grated opening in the mosque leading to the cave. Childless women would petition Sarah for fertility (Sarah got pregnant when she was 90), pilgrims would feed Abraham bark from his famous oak in Mamre, and so on (Dumper 2020, p. 165). Even non-Muslims could petition the spirits by bribing the guards.

Auerbach recounts many more fascinating legends about the cave.

Perhaps a Jew and a Muslim had a conflict and the Muslim suddenly died? Someone had seen the Jew near the mosque and the locals "put two and two together" as superstitious people often do. So they asked the sultan not to let Jews (or Christians) closer than five steps on the mosque's stairway entrance. But for some reason that wasn't enough so they had to extend it to seven steps. Maybe the Muslims observed Jews prostrating themselves against the stones of the walls when praying and reasoned that if the Jews wanted to get as close as possible, keeping them at a distance would be safer?

This is of course just a theory, but since it is not known why the ban was imposed we can't know for certain why Jews weren't allowed closer than five or seven steps.

• Michael Dumper (2007). Cities of the Middle East and North Africa: A Historical Encyclopedia
• Jerold S. Auerbach (2009). Hebron Jews: Memory and Conflict in the Land of Israel