What is the origin of the belief that Satan tortures the unsaved in Hell?

If not, what is the origin of that belief?

The Catholic Church teaches that God is not only merciful [cf. e.g. CCC 1990 and CCC 1992], he is also just [cf. CCC 2009]. And God being just, when each and everyone one of us will appear before the judgment seat of Christ, each will receive good or evil, according to what he has done in the body. [Cf. 2 Cor 5:10 (RSVCE)]

Therefore the Church teaches that for some there will be heaven for ever where God will bestow happiness and the richest gifts on all those who depart this life free from original sin and personal mortal sin, and who are, consequently, in the state of justice and friendship with God, and hell for ever for those who die in personal mortal sin, as enemies of God, and unworthy of eternal life, they will be severely punished by God after death.

Therefore the Church teaches the "punishment" of the damned vs. the "torture" of the damned.

The question then becomes whether we can find within the Church the notion of the punishment God inflicts being carried out by the devils, and the answer is yes according to Dialog of Catherine of Siena: How even the devils render glory and praise to GOD:

“Sinners, such as those of whom I have just spoken, are placed in this life in order to augment virtues in My servants, as the devils are in Hell as My justiciars and augmenters of My Glory; that is, My instruments of justice towards the damned, and the augmenters of My Glory in My creatures, who are wayfarers and pilgrims on their journey to reach Me, their End. They augment in them the virtues in diverse ways, exercising them with many temptations and vexations, causing them to injure one another and take one another’s property, and not for the motive of making them receive injury or be deprived of their property, but only to deprive them of charity. But in thinking to deprive My servants, they fortify them, proving in them the virtues of patience, fortitude, and perseverance. Thus they render praise and glory to My Name, and My Truth is fulfilled in them, which Truth created them for the praise and glory of Me, Eternal Father, and that they might participate in My beauty. But, rebelling against Me in their pride, they fell and lost their vision of Me, wherefore they rendered not to Me glory through the affection of love, and I, Eternal Truth, have placed them as instruments to exercise My servants in virtue in this life and as justiciars to those who go, for their sins, to the pains of Purgatory. So you see that My Truth is fulfilled in them, that is, that they render Me glory, not as citizens of life eternal, of which they are deprived by their sins, but as My justiciars, manifesting justice upon the damned, and upon those in Purgatory.”

In the New Advent article on hell above, among the accidental pains of the damned are that the reprobate must live in the midst of the damned [devils included]; and their outbursts of hatred or of reproach as they gloat over his sufferings, and their hideous presence, are an ever fresh source of torment.

Conclusion

Perhaps it is from the notion that devils execute God's punishment in hell, where the damned are tormented, that some take this to mean Satan/the devils torture the unsaved in Hell.

Endnote

Jesus spoke of Satan's kingdom and of the devil and his angels. He also Jesus taught that whoever commits sin is a slave of sin, some of which are committed under the instigation of Satan or the devils and Scripture in Acts teach how Jesus was healing all that were oppressed by the devil, it is therefore not farfetched to think of Satan as the lord of hell [cf. Beelzebul | Jewish Encyclopedia].

Lucifer, King of Hell; Canto XXXIV of The Inferno, by Dante Alighieri; Illustration by Gustave Doré

There are two pieces of the puzzle here: First Hell, and Second, Satan. The question is when were these two pieces first merged together. Technically speaking, the Bible never explicitly links these two. Despite this, there is an impression - even in Jesus teachings - that the most evil creature ruled over the underworld in all religions back to the earliest known records - including Christianity.

Hell

In Canonical Biblical writings, the concept of the underworld (שְׁאוֹל; Sheol) first appears in Genesis 37:35

All his sons and daughters stood by him to console him, but he refused to be consoled. "No," he said, "I will go to the grave [שְׁאוֹל; Sheol] mourning my son." So Joseph's father wept for him.

And in Job 26:6, this place is regarded as either being or containing a place of destruction.

The underworld is naked before God; the place of destruction lies uncovered.

Much like in Christianity, where some believe judgement is not immediate and the soul exists in an intermediate state before the Resurrection and the Millenial Kingdom, the same is true of some Jewish schools of thought where souls are though to dwell in שְׁאוֹל (Sheol) until "Olam Ha-Ba" (the world to come) at which point there will be a bodily resurrection of the dead.

Similarly, where some in Christianity believe judgement is immediate and the righteous go to Heaven to dwell with God before the final judgement at the end times in Revelation, some schools of Jewish thought believe that the righteous go to Gan Eden upon death. Where Christians most often believe that the unrighteous go to Hell upon death, some according to Jewish thought go to Ghenna/Gehinom. Depending on the viewpoint, this may be an eternal punishment, or may only be punishment for a time, similar to the Catholic doctrine of Purgatory.

In any event, Luke 16:19-31 paints a pretty vivid picture of Hell being a place of torment.

Satan

The word Satan derives from the Hebrew word שׂטן for "accuser" or "Adversary" used in Job to describe the character that many regard as Satan. The Hebrew and Aramaic Lexicon of the Old Testament defines this word as,

שָׂטָן
1 accuser, adversary: a) human: 1K 5:18; b) mal˒ak yhwh + Nu 22:22, 32; — 2. spec. supernatural figure: haśśāṭān, the Satan + Zc 3:1f Jb 1:6–2:7 (14 ×); > śāṭān (proper name) + 1C 21:1

Modern Judaism does not regard Satan as being malevolent or inherently evil, but instead simply being a servant of God to ensure that mankind has free will.

Appearing alongside the entry for HaSatan in the HALOT is this entry however,

שׂטן
qal: impf. יִשְׂטְנוּנִי; inf. שִׂטְנוֹ; pt. pl. cs. שׂטְנֵי, sf. שׂטְנַי: bear a grudge against, harbor animosity toward Ps 38:21.

Accordingly, Christianity regards Satan as malevolence and the epitome of evil because if God is truly good, then the opponent and adversary must be opposed to that good - therefore evil.

Judiasm around the time of Jesus also paints this picture with books like 1 Enoch painting Satan as a evil, malevolent fallen angel.

Mark Edward and I were also able to trace the link between Satan and the Serpent in the Levant all the way back to the earliest Egyptian inscriptions. Genesis appears to have been written with the understanding that ancient readers would be comparing the Genesis creation account to Egyptian and Babylonian creation accounts.

Nearly all of the creative stories in the ancient Middle East had a strong serpent motif with serpents representing evil. This is especially pronounced in Egyptian and Greco-Roman mythology with the serpent Apep ruling the underworld - a place of chaotic torment in Egyptian mythology. Every day, Apep was thought to torment Ra as he traveled through the underworld. By inviting a comparison to these myths, the Bible (Genesis) is essentially equating and implying that the serpent and Satan are an evil tormentor who rules the underworld.

Conclusion

In Egyptian, Greco-Roman and most other mythology, the god who ruled over the underworld was typically the embodiment of evil - usually one of the more evil if not the most evil god in the pantheons. Wikipedia states,

Gehenna was initially where apostate Israelites and followers of various Ba'als and other Canaanite gods, including Moloch (or Molech), sacrificed their children by fire (2 Chr. 28:3, 33:6). Thereafter it was deemed to be cursed (Jer. 7:31, 19:2-6)

Yahweh is a powerful God. Therefore, anyone capable of standing in opposition to Him must also be powerful. As Gehenna is the destination of evil, it is only natural to assume that that power would allow the adversary to rule over Gehenna (hell). When we remember that all of the other creation stories had the token malevolent god ruling over the underworld and realm of the dead, it is easy to see how the belief that the adversary was ruler of the place of torment and that the adversary was the primary tormentor himself (as lord of that realm) came to be.

Therefore, while there may not be an explicit Biblical basis for this idea, the idea is certainly implicit to the text and the traditions surrounding it. Furthermore, the origin of this belief can be traced to the earliest known records of any religion.