Was casting out of demons done in the Old Testament by any Prophets or was it started in the New Testament?

Per Caleb's response:

There are no explicit references to exorcism in the OT. However, in a possible foreshadowing of Christ, David caused an evil spirit to depart from Saul multiple times by playing the harp (1 Sam 16, 18, 19).

I wanted to point out that some Hebrew scholars rightfully point out that the word for 'evil' is 'rah' which actually has many similar definitions that are not what we think of when we say 'evil'. A cursory search turns up the following definitions:

• bad, evil
• evil, distress, misery, injury, calamity

So, potentially, there could be no incidences of demon possession in the old testament. To be fair, there are many negative, supernatural situations in the OT but they are incidences where Satan intervenes or the Israelites are cursed by God for breaking their covenant with Him. But again, no outright demonic possessions. I stumbled across this stackexchange question while researching against Matthew 9:32-34 where Jesus heals a demon-possessed mute and the people exclaim “Nothing like this has ever been seen in Israel.”

Servants of Israel's God cast evil spirit through spiritual songs during the Old Covenant. Demon possession were rare because they were under God's protection until their transgression. Jewish exorcists Jesus mentioned were using the Qabbalah, or Jewish magic, in casting out demons. But true exorcism is only through commanding words, spiritual songs, fasting and prayer. And not through rituals as we can see from all world religions. That makes Judeo-Christian different above the rest.

Matthew 12:28

But if I {Jesus speaking} cast out devils by the Spirit of God, then the kingdom of God is come unto you.

This implies that Jesus was the first to cast out devils in this manner.

No one in the Old Testament ever cast out devils like Jesus did. Jesus was signing His signature as being the Son of God having all power and authority over all spirits, clean or unclean.

In my own imagination, to perform "casting out a demon from someone's inside" must first started to the thinking that someone is possessed by a demon.

Because the question is not asking from a specific denomination, my answer is from outside source which I just read from this site. The article said that it seems how the Jews view about demon is changing from the Old Testament time until (at least) a few hundred years after Jesus time.

Below is what article said about the view of Israel of demon during OT time

The Israelite conception of demons, as it existed in the popular mind or the literary imagination, resembled in some ways that held elsewhere. Demons live in deserts or ruins (Lev. 16:10; Isa. 13:21; 34:14). They inflict sickness on men (Ps. 91:5–6). They trouble men's minds (Saul; I Sam. 16:15, 23) and deceive them (I Kings 22:22–23). – but nevertheless these evil spirits are sent by the Lord

The article continue :

A great change had taken place in angelology and demonology, at least in certain circles within Judaism, by the last centuries B.C.E. In this period the religion, while safeguarding its monotheistic character in various ways, nevertheless took on many traits of a dualistic system in which God and the forces of good and truth were opposed in heaven and on earth by powerful forces of evil and deceit. This seems to have been under the influence of Persian religion, with its opposition of Ormuzd the good god and Ahriman (Angra Mainyu) the evil god, but at the same time Jewish dualism drew on older, native resources in constructing a more elaborate demonology

I think that is a transition period (on how they view about demon) to the time when Jesus time.

According to the article, below is Jewish view about demon around Jesus time

New Testament demonology in part reflects contemporary popular belief, which turns up also in rabbinic literature, and in part the dualism attested in the sectarian literature from Qumran. Demons are called "unclean spirits" or "evil spirits," as in rabbinic literature. They are believed to inhabit waste places. Possession by demons causes, or is associated with, various sicknesses, especially those in which there is a perversion of the human personality, so that the demon, not the man himself, directs his acts and speech (Mark 1:23, 26; 9:17–29)

It's difficult to know when the Jews starting to believe that someone is possessed by evil spirit, because it's possible at the time Jesus perform the casting out then they started to know that a demon is able to possess someone, which later on they follow Jesus act... but from some sentence in a verse, for example

Luke 8:29
For He had commanded the unclean spirit to come out of the man. For it had seized him many times; and he was bound with chains and shackles and kept under guard, and yet he would break his bonds and be driven by the demon into the desert.

I think before any "casting out a devil" for the first time performed by Jesus, they already believe that someone can be possessed by a demon. Hence, at the time of Jesus was born, maybe they already starting to try to cast out a demon from someone.

Was casting out of demons done in the Old Testament by any Prophets or was it started in the New Testament?

As some posts here already said that from reading the OT it's difficult to draw a conclusion to answer the question, assuming what article said about their view of demon is correct, then most likely (I think) - there is no "casting out demons from inside someone" performed by the Prophets at that time.

If you mean "started in the NT" is after for the first time Jesus perform the casting-out, I think when Jesus was born they already started (or at least trying) the casting-out, assuming their view about the demon already "someone is possessed by a demon".

There could be none in the Old Testament to show that someone is casting out demons, and the reason is simple, in Genesis, God created man and gave authority on all animals and plants but not spirit.

When Jesus came, He gave men authority to cast out demons in His name.

And that is a powerful confirmation of His Glory!!! Glory to God!

There seems to be at least two vague incidents in Scriptures for those who are of Catholic and Orthodox traditions.

The first incident involves David where he plays the lyre/harp (1 Sam 16: 18-23) in Saul's presence and the Spirit of Melancholia could flee his presence.

Whenever the spirit from God came on Saul, David would take up his lyre and play. Then relief would come to Saul; he would feel better, and the evil spirit would leave him (1 Samuel 16: 23).

Fr. Gabriele Amorth SSP mentions in this in one of his books that the singing of sacred hymns is a valid form of expelling the demon. Being a Catholic priest moreover, he encouraged the singing of Gregorian Chant during exorcisms, since Gregorian Chant in itself is very scripture based. Fr. Amorth makes note of David playing the harp incident.

The second incident comes from the Book of Tobit which is part of both the catholic and Orthodox biblical canons. The Book of Tobit recounts the story how the young Tobias with the aide of the angel Raphael made the demon flee their presence.

When they had finished eating, they escorted Tobi′as in to her. As he went he remembered the words of Raphael, and he took the live ashes of incense and put the heart and liver of the fish upon them and made a smoke. And when the demon smelled the odor he fled to the remotest parts of Egypt, and the angel bound him (Tobit 8).

Fr Gabriele Amorthe mentions in one of his two books (on exorcisms) this incident and encourages the use of blessed incense in homes subject to demonic influence.

I don't think there is any other reference to casting out out of demons other than what's stated above about David. BUT Jesus said this in Matthew 12:27 NLT

And if I am empowered by Satan, what about your own exorcists? They cast out demons, too, so they will condemn you for what you have said.

So it seems as if casting out of demons already was practiced before Jesus started doing so.

There are no explicit references to exorcism in the OT. However, in a possible foreshadowing of Christ, David caused an evil spirit to depart from Saul multiple times by playing the harp (1 Sam 16, 18, 19). Some view the NT exorcism accounts as evidence of an increased attempt by Satan to thwart God's plan to redeem the world through Jesus.