What was Muhammad's religion before becoming a Muslim prophet?

Hanif is not in fact a religion. Hanif is a description for a believer. any kind of believer having any divine religion like Christian or Muslim can be a Hnif believer.

This can be seen in Quran. when Quran calls the prophet Ibrahim a Hanif Muslim:

Abraham was neither a Jew nor a Christian, but he was one inclining toward truth, a Muslim [submitting to Allah]. And he was not of the polytheists. http://tanzil.net/#3:67

In this verse the two words Hanif (حنیف) and Muslim (مسلم) are used for describing prophet Ibrahim a.s. this translator has translated the word Hanif to "one inclining toward truth".

According to Islamic sources although most of Arabs were polytheists but prophet Muhammad SAWW and his fathers (who are children's of prophet Ibrahim a.s. and Ismael a.s.) never have been polytheists and all have been monotheists. all the genealogy line from Ibrahim a.s. to Muhammad SAWW have been believers and monotheists.

Please note although Muhammad SAWW started preaching Islam at age 40 but this does not mean he himself did not know anything about Islam before it. he was practicing Islam from when he was child and never committed any sin all over his life and if he did not preach Islam publicly before 40 it was for political conditions of that time and if he preached Islam before the suitable conditions and before finding enough secret supporters he would be killed by polytheist state King and state and culture and people of that time and so never could preach Islam. when he started preaching he and his supporters were kept under sanctions and were enforced to live out of city without have any right to buy or sell anything and the polytheists tries to kill him many tiems and after having many fights and wars with polytheists who their boss was Abu Sufyan finally Muslims after many years could conquer Mecca the base of polytheists without war.

He himself said do not follow my ancestors before than Adnan (the 20th ancestor) and connect it to Ismaeel (the son of prophet Ibrahim) because what Arabs say about before Adnan is not correct.

Reference:

Ancestors of prophet SAWW

To add to the answer's already here:

Muhammad would have probably been brought up in the pagan culture of his tribe.

However, according to accounts, his job as a young man was to lead his uncle's merchant caravans. This would have meant travelling from Mecca to other big trading spots in the area, like Damascus for example. This would have exposed the young Muhammad to Abrahamic monotheism, namely Christianity and Judaism (indeed I believe the hadiths mention that Muhammad's encounter with a Christian monk had a great influence on him, don't quote me on this though).

The OP asked for historical accounts, unfortunately there is unlikely a complete and accurate first hand account Muhammad's life before his prophet-hood, including about his belief. According to Karen Armstrong, "We know practically nothing" about Muhammad's life before his receiving of revelation (and becoming a prophet), because he was not yet a major figure at that time and people did not think it worthwhile to record about him.

Only during the last ten years of his life, after he became a major political leader and people around him realized that history was being made, his life was recorded in much more details.

Firstly, there's no documentation or archaeological data available regarding the matter in question. Arabs of that time were mostly illiterate. There were extremely few individuals who could read and write. The culture of the period was almost entirely based on oral transmission.

All of the reports about his beliefs before prophethood is based on Islamic traditions. These traditions indicate that he was a member of the Arab tribe Banu Hashim who practiced monotheism. Al-Islam.org writes:

There was a small group of monotheists present in Arabia on the eve of the rise of Islam. Its members did not worship idols, and they were the followers of the Prophet Abraham. The members of the families of Muhammad, the future prophet, and Ali ibn Abi Talib, the future caliph, and most members of their clan – the Banu Hashim – belonged to this group.

Muhammad was a member of the Quraish tribe, which controlled the Kaaba in Mecca. Prior to Islam, the Kaaba was an important pagan pilgrimage site. So that suggests at least what his cousins and extended family may have believed.

Today the Kaaba is the official direction of Muslim prayer and all mosques and prayer locations will have a qibla to orient the supplicant towards the Kaaba (in some Western countries, the qibla is sometimes shorthanded by non-Muslims as pointing "east"). But prior to being the Muslim center of the world, it was a pagan site for centuries.

The Quraish took control of the site generations before Muhammad, and profited from that position. The Quraish generally followed a polytheistic pantheon. The Quraish wiki article quotes The Oxford Handbook of Late Antiquity:

The Qurayshite pantheon was composed principally of idols that were in the Haram of Makka, that is, Hubal (the most important and oldest deity), Manaf, Isaf, and Na'ila.

The wiki article also summarizes Abdullah Saeed, The Qur'an: An Introduction as stating that the Quraish had a pantheon of one higher God with multiple lesser Gods.

Note that the Quraish also formed the early opposition to Muhammad. The earliest stages of the religion were a quarrel over possession of Mecca and the Kaaba, so naturally the more powerful elements of Quraish opposed Muhammad, who was not from the most powerful elements of the tribe. He fled to Medina but eventually defeated Mecca and pardoned his tribesmen, who became early Muslims.

I'm not sure how much that tells you about what historical Muhammad may have believed, but it gives you some sense of the broader social environment.