Why are there distinct Orthodox and Catholic Christian communities in Bosnia and Hercegovina?

The official religion in the Balkans was Calcedonian Christianity during the later Roman Empire. Then, about AD 600, the region was invaded and settled by pagan Slavic tribes.

Naturally both the eastern church headquartered at Constantinople and the western church headquartered at Rome occasionally sent out various missionaries to try to convert the Slavic tribes in the Balkans. The eastern Roman of "Byzantine" Empire also sought to reconquer the Balkans.

Nomadic Bulgars settled in the Balkans and mingled with the Slavic tribes. The Bulgars converted to eastern and Orthodox Christianity and conquered most of the Balkans and converted most of the Slavs to the eastern type of Christianity. The First Bulgarian Empire became a mighty rival of the eastern Roman or "Byzantine" Empire.

The tribes who lived in the most northwestern part of the Balkans adopted western, Roman Catholic, Christianity and the Latin Alphabet and became known as Croats. The similar tribes who lived in the south central parts of the Balkans adopted eastern, Orthodox, Christianity and the Cyrillic alphabet based on the Greek alphabet, and became known as Serbs. The spoken language of Serbs and Croats is called Serbo-Croatian. Note that the Slavs living in Bosnia, between Croatia and Serbia, would probably also speak Serbo-Croatian, and modern Bosnian is a form of Serbo-Croatian.

[04-20-2019. Because closely related tribes came from the north and settled the neighboring regions that became Croatia, Bosnia, and Serbia, Croats, Bosnians, and Serbians have always been culturally similar, and it was easy for members of one of those groups who settled in a region occupied by another similar group to become culturally assimilated over generations and centuries]

Eventually the eastern Roman or "Byzantine" emperor Basil II "The Bulgarslayer" completed the reconquest of the Bulgarian empire and all of the Balkans in 1018. The "Byzantine" empire naturally continued to make Orthodox Christianity the official religion in all the Balkans.

The final official spit between the Roman Catholic and eastern Orthodox Churches came in 1054 after centuries of increasing differences.

The Kingdoms of Croatia and Hungary were joined in a personal union after about 1100, giving Hungary an interest in the Balkans.

Stefan Nemanja of Serbia revolted against the "Byzantine" Empire after 1180, and in 1185 the Bulgars revolted against the "Byzantine" Empire and founded the Second Bulgarian Empire. In 1204 the 4th Crusade captured Constantinople and the "Byzantine" Empire dissolved into many smaller states.

For decades greater powers such as Epirus/Thessalonika, The Latin Empire, Nicaea, Serbia, and Bulgaria competed for control of the Balkans and smaller states were conquered or vassalized by the larger ones. And the more powerful kings of Hungary and Croatia often extended their power and influence into the Balkans. Many Balkan regions were sometimes vassals of Hungary.

Note that Bosnia was in an angle between Croatia and Hungary, which had the same king during the Middle Ages, and so was made a vassal of a powerful Hungarian ruler first and most often during the Middle Ages. And Bosnia was right at the border of Serbia and quick to ally with or become a vassal of the more powerful Serbian rulers.

So some Bosnian Christians joined the national Bosnian church, but others became Roman Catholics like the Hungarians and Croatians, while others became Eastern Orthodox like the Serbians. And over the centuries for various reasons many Croats and Serbs settled in Bosnia, where the native language would have been Serbo-Croatian and thus easy for them to understand.

The above information is basically Balkan history 101, a very brief and elementary history of the Balkans in general and Bosnia in particular.

Added 04-20-2019.

The nature of the Bosnian church is little known today.

Roman Catholicism and Eastern Orthodoxy initially predominated in different parts of what is today Bosnia and Herzegovina; the followers of the former formed majority in the west, the north and in the center of Bosnia, while those of the latter were a majority in most of Zachlumia (present-day Herzegovina) and along Bosnia's eastern border (from Drina to Vrbas river)

https://en.wikipedia.org/wiki/Bosnian_Church1

A separate Bosnian church was established in 1252 when the Bosnians refused to be put under the authority of a Hungarian bishop. Many Bosnians remained members of the Roman Catholic or Eastern Orthodox churches.

This separate Bosnian church existed for about 200 years, although it is uncertain whether it ended before, during, or after the Turkish conquest in 1463. It probably gradually faded away as its members became more and more Roman Catholic or Eastern Orthodox to fit in with local communities.

https://en.wikipedia.org/wiki/Bosnian_Church1

The Bosnians were often accused of being heretics, like the Bogomils of the Balkans and their Cathar offshoot in southern France. The bloody and genocidal Albigensian Crusade of 1209 to 1229 against the Cathars showed how dangerous it was to be accused of Bogomilism.

In fact popes preached several crusades against Bosnia, one canceled in 1203, and two in 1221 and 1225 that never got off the ground. Another crusade against Bosnia was called in 1234, resulting the Bosnian Crusade of 1235-1241, a Hungarian invasion of Bosnia that only ended when the Mongols invaded Hungary in 1241.

https://en.wikipedia.org/wiki/Bosnian_Crusade2

No wonder many Bosnians were willing to break with the Roman Catholic Church in 1252.

So called Bosniak nation is recent invention, from 1990's

First you must understand genesis of Muslims in Bosnia. They are essentially mixture of Slavic population that existed before Ottoman invasion, and various other Islamic ethnic groups that came with invaders ( Turks, Albanians, Arabs, Roma (Gypsies) etc ...) . Before aforementioned invasion, population of Bosnia was Christian. But, do to isolation and remoteness of Bosnia, official church (either Orthodox or Catholic ) was not that strong. In fact, for long time there exist Bosnian Church, considered heretical by both Catholics and Orthodox. Although it did officially disappear before Ottoman conquest of Bosnia, it influence was still strong in a sense that significant part of population only nominally followed official religion of the ruler (either Catholic or Orthodox) . Note that Bosnian kings often did change their religious affiliation when it suited them (from being Catholics as vassals of Hungarians, to being Orthodox as relatives of Serbian rulers) . Dukes of Saint Sava on the other hand were of course firmly Orthodox and therefore much more closely connected with Serbian medieval state and tradition.

Anyway, with Ottoman conquest large percent of population decided (willingly or under coercion) to switch to Islam (do to the fluidness of religion in Bosnia, much larger percent then for example in Serbia) . What is important to understand is that they considered themselves as Turks and their faith as Turkish faith. Even their former compatriots considered them as such. In oral and written tradition from medieval times till 19th century they are called Turks, or domestic Turks (Turci , domaći Turci in Serbian). Their faith is called Turkish faith (turska vera) and their law is called Turkish law (turski zakon). Sometimes the term Turk convert (poturica in Serbian) was used by Christians to denote this part of population. Obviously, Christians and Muslims in Bosnia were in state of conflict for centuries, and their communities at best lived beside each other, never together. This is important to understand further developments.

With the end of Ottoman rule in 19th century, Bosnian Muslims did find themselves in predicament. Their "Turkish" identity was gone, they had language that was essentially same as Serbian or Croatian, they had same biological origins as their neighbors (some of more prominent families even remembered when they converted to Islam, and who was their Christian patron saint before that), and only thing that separated them was their religion. Austria-Hungary which occupied Bosnia after Ottomans skillfully used this religious divide for their advantage (divide and rule) . Since Orthodox already had distinct Serbian identity, they cultivated Catholics as Croats and kept Muslims separated from both. In such manner they had three distinct nations in Bosnia which were easier to rule then one unified nation.

There were attempts to declare Bosnian Muslims as simply Islamic Serbs or Croats, but these efforts mostly failed, because of completely different culture, history and tradition. For example, what would be war for liberation from Ottoman yoke for Serbs, would be disaster for Muslims. Croatian puppet state in WW2 even declared Muslims as "flower of Croatian nation" but this simply could not stick.

During socialist Yugoslavia, Bosnian Muslims become a nation (Muslimani - Muslims with capital M ). This designation held until the beginning of breakup of Yugoslavia and Bosnian Civil War, even for those members of that nation who were not particularly religious . After the conflict started, it became clear for Bosnian Muslim leadership that insisting on such designation would counterproductive for their war aims of creating unified Bosnia and Herzegovina . Therefore, they started using terms as "Bosniak" or "Bosnian", while their enemies continued to call them Muslims or even Turks .

Finally, there needs to be certain linguistic clarification about terms Bosnian (Bosanac in Serbian ) and Bosniak (Bošnjak in Serbian) . Both of them (especially first, second is more archaic) originally detonated man from Bosnia as region, and did not imply religion. Therefore Serb could be Bosnian, Croat could be Bosnian and Muslim could be Bosnian. After the war, those terms started to detonate exclusively Muslims. Therefore presently Serb from Bosnia would rarely describe himself as Bosnian (Bosanac) and practically never as an Bosniak (Bošnjak). Same goes for Croats. In fact, whole country (Bosnia and Herzegovina) is presently held together only by Western pressure, Muslim parts are becoming more Muslim everyday (check for example recent photo of high school graduates in Tuzla ) and Christians simply do not want to live with them, or share Bosnian identity.