If Buddha stayed unanswered on this matter, why do his followers preach that soul doesn't exist?

Buddhism believes in the mindstream (santana), but the mindstream is not a thing (atta). This is the difference. The Dalai Lama has a little book called Buddha Nature that explains this perfectly.

What is meant by "non-existent" is that it doesn't exist in the way we naturally presume it exists (inherently existent). Instead, the "self" exists as all conditioned phenomena, comprised of various causes and conditions i.e it is dependently originated.

A useful guide for those who are merely interested of how those concepts are used on Buddhas path rather to take any stand: Selves & Not-self Of course it will not prevent those who are seeking for stands to nourish their stand. As a Ajahn once said: "To be or not to be. That is no question." It makes you simply to become again and again.

Once you get into the concept of the soul in Buddhism, you have essentially entered into Advaita Vedanta of Hinduism. India largely was converted to Buddhism at one point, which meant they were rejecting the Vedas, the holy scriptures of Hinduism. From a Hindu perspective, the Buddha incarnated for the specific purpose of rejecting the Vedas, because they were being abused by the priests and the teachings were being distorted. So the Buddha incarnated and presented a path in which it cultivated all the qualities necessary for nirvana, without dependence on the concept of God and Atman, which is essentially how people related to the Vedas. Then Adi Shankaracharya came to India and argued against Buddhist philosophy, by asking who is the one seeking nirvana? See https://www.youtube.com/watch?v=63TPsywU5Rw

His arguments restored faith in the Vedas, and that is why India is predominantly a Hindu culture again. His teachings are essentially Buddhism with the Atma. I am not saying that you have to accept these arguments. Ultimately your spiritual path should be defined by where your heart is, and where you find your master. There is nothing wrong with investigating this path of Advaita Vedanta. Both are very powerful traditions that generally respect each other, and lead to the same results at the end. From a beginner's perspective they can seem highly different, but the longer you stay in one path you will realize that they are leading to the same place. Whichever path you choose, you need to find a genuine teacher who embodies the teachings completely, and is a manifestation of compassion.

Philosophy of reality is only as important as it brings us to knowledge. Buddhism is just saying like you say, the Atma is not important. Meditate, be compassionate, that is the emphasis in Buddhism. Advaita Vedanta holds meditation to be very important, but it also places an emphasis on cultivating devotion to the divine in order to realize the Atma and the divine are nondifferent. Ultimately, the experience of Nirvana and Brahman I believe are they same. Krishna in the famous Bhagavad Gita refers to nirvana in that exact word, prior to the Buddha. You will see so much similarity in Buddhism and Hinduism, even some of the deities are practically identical, they are just drawn differently because of a different culture. They both use sanskrit and puja to deities. No need to worry or be upset about this. You can gain success in either way. Find the way where you experience the most love and devotion, that is your path. They are both excellent, respected ways. Just be careful, there are false gurus in either path.

Ultimately, the truth will be found through cultivation of dharma, i.e. good qualities, abstaining from harming others, and sincere meditation.

The Buddha did not remain silent on this matter. He only remained silent to those who would misunderstand. As stated in the Ananda Sutta, the questioner of the Buddha was "befuddled" & "confused". To quote: "the bewildered Vacchagotta would become even more bewildered".

As for 'nihilism', it is a 'self-view', which believes "I" will die or "others make me suffer". Relevant suttas that explain this are: Brahmajala Sutta, Held by Views (Itivuttaka) and Acela Sutta (refer to footnote).

This is an ancient debate that will not get settled today.

The problem is that any self we can identify automatically takes on - in our mind's eye - qualities like permanence and continuity. These qualities are false.

On the other hand - saying there is no self at all is also clearly false. It was a rare kind of nihilism to fall into in Buddha's day, but in a community like this it's more likely.

Anyhow: you´ll have to study the various schools of Buddhist philosophy to get a handle on the many ways this question has been answered.

You may want to start here: http://contemporarybuddhism.com/cyclic-existence-emptiness/

Asking if "is there a soul in me?" or "is there not a soul?" are questions coming from a wrong framework to begin with. It's like asking "Does batman have a wife or does he not?". This question is invalid in the real world, because Batman only exists in the DC universe.

Also, when you ask "is there a soul?", you have to first clarify 'is'. Are you referring to form? or is it feelings? or perception, mental formations, awareness? According to the Anatta Lakkhana Sutta, none of those qualify as a self. See if you can experience any reality that doesn't fall into those 5 categories?

But what about the practitioner himself who is scanning or performing the vipassana? Doesn't that imply that he himself is the soul?

What is scanning? It means awareness. If awareness is a self, you should be able to be aware of things at will. Why do you need to develop Sati? Why does awareness disappear without your consent? Why can't you be aware of sights without the eyes? Why can't you be aware of sound without the ears? So is awareness worthy of being considered as a self?

Finally, the Buddha not answering the question in certain instances does not imply that there is a soul. It could be because answering it could have confused the person even more. Anatta Lakkhana Sutta was preached to Sotapanna, Anagami theros. They were well into the path and had little dust in their eyes. It is certainly not a topic for the average layman and not the best entry point to Buddhism for a beginner.

The Buddha did explicitly refute the notion of a permanent self in the Ananda Sutta. He says that if he had said that there is a permanent self then that would conflict with the statement that "all phenomena is not self".

"Ananda, if I — being asked by Vacchagotta the wanderer if there is a self — were to answer that there is a self, that would be conforming with those brahmans & contemplatives who are exponents of eternalism. If I — being asked by Vacchagotta the wanderer if there is no self — were to answer that there is no self, that would be conforming with those brahmans & contemplatives who are exponents of annihilationism. If I — being asked by Vacchagotta the wanderer if there is a self — were to answer that there is a self, would that be in keeping with the arising of knowledge that all phenomena are not-self?" "No, lord." "And if I — being asked by Vacchagotta the wanderer if there is no self — were to answer that there is no self, the bewildered Vacchagotta would become even more bewildered: 'Does the self I used to have now not exist?'"

The concept of eternalism says that there is a permanent self and it is undying. The concept of annihilationism says there is no self at all and at death it all ceases.

Buddha's teaching is of the middle way stating there is a self that "emerges", as a result of the dependency on the inter-working of the five aggregates of form, feeling, perception, mental formations and consciousness. Each of these five aggregates also have dependencies. This is what is meant by the Buddha's statement, "all conditioned things are impermanent", in other words, all these things that have dependencies are not permanent.

However, there is no single self that is the charioteer or agent of the five aggregates. Imagine you hurt your hand and say "I hurt myself". But if your hand is chopped off, does your self disappear? No. So, applying this one by one, you realize that your self is neither form, feeling, perception, mental formations or consciousness. But beyond this, there is nothing more to yourself. That is what is meant by the Buddha's statement, "all phenomena is not self".

This is just an overly-generalized summary. More info can be found on the topic of dependent origination.

Now according to the concept of annihilationism, there should be no continuity at all and no rebirth, but there is. If there is no permanent self then what is it that is reborn? The idea is that "information" or chain of dependent processes that never ceases, and keeps reappearing, so long as the desire and attachment to exist does not cease. This is described here (Milindapanha 3.5.5):

The king asked: "Venerable Nagasena, is it so that one does not transmigrate and one is reborn?" "Yes, your majesty, one does not transmigrate and one is reborn." "How, venerable Nagasena, is it that one does not transmigrate and one is reborn? Give me an analogy." "Just as, your majesty, if someone kindled one lamp from another, is it indeed so, your majesty, that the lamp would transmigrate from the other lamp?" "Certainly not, venerable sir." "Indeed just so, your majesty, one does not transmigrate and one is reborn." "Give me another analogy." "Do you remember, your majesty, when you were a boy learning some verse from a teacher?" "Yes, venerable sir." "Your majesty, did this verse transmigrate from the teacher?" "Certainly not, venerable sir." "Indeed just so, your majesty, one does not transmigrate and one is reborn."

"Especially, the vipassana practitioners say that scanning the body and finding no atman or self in it is proof that self doesn't exist."

The reasoning of said practitioners is flawed -- otherwise, I could claim absurdities like "there is no sun" after scanning my body and not finding a burning star in it.

"But what about the practitioner himself who is scanning or performing the vipassana? Doesn't that imply that he himself is the soul?"

If you regard soul to be a "fixed, non-changing, eternal self, ...", it doesn't. The implication would only hold if you'd show that "it" is fixed, non-changing, eternal, under your complete control ("may it be thus", "may it not be thus"), and not a source of dukkha. These are the characteristics often ascribed to atman.

"So, how can you argue that soul doesn't exist when Buddha himself didn't deny it (or accept it either)?"

More was said about this:

“Bhikkhus, you may well cling to that doctrine of self that would not arouse sorrow, lamentation, pain, grief, and despair in one who clings to it. But do you see any such doctrine of self, bhikkhus?”—“No, venerable sir.”—“Good, bhikkhus. I too do not see any doctrine of self that would not arouse sorrow, lamentation, pain, grief, and despair in one who clings to it.

-- Alagaddūpama Sutta, MN 22

"Are there any branches of Buddhism that believes in existence of soul?"

I've read about some old schools that did so (but not without going through some philosophical hoops to avoid clash with the suttas and inspiring rebuttals from other schools), however I don't quite remember and one has to dig a little to see who they were and what was their doctrine (whose texts may not have survived to this day). I don't know of any current school who advocate atma doctrine.