How did Japan build an extensive rail network in Greater Tokyo?

I found the direct answer. (To be updated later).

Conclusion

Yes, the government bought the houses and demolished (by today's JR East).

Hereunder the text under the quote in bold italic is the translation.

From this statement of the Board Of Audit Of Japan.

日本国有鉄道大阪幹線工事局で、東海道幹線増設工事に伴い、京都府乙訓郡大山崎村所在の福田某所有の宅地、建物等の買収費および移転補償費として、昭和３７年４月および５月、同人に総額２１５，０１０，４２０円を支払っているが、当該増設工事に要する用地は宅地の一部にすぎないものであるのに、全宅地、建物等について買収等をしたのは処置当を得ないと認められる。

First, you need to understand that before 1987, all the railways were nationalized. (So-called Kokutetsu).

Regarding the kokutetsu Osaka construction department's payment to Mr.Fukuda who resides at Ooyamazaki village Otokuni county, Kyoto, which totals 15,010,420 yens on April and May 1961 in order to extend the Tokaido bullet train lines(operated by today's JR East) was not appropriate because the land the construction department bought was part of Mr.Fukuda's residence, only minimal, but the department paid the amount as a compensation for Mr.Fukuda's to move his whole residence

So, yes, Kokutetsu, aka today's JR east, paid the compensation fee in order to extend their lines.

Thank you.

It appears that the answer to this question has more to do with Japans history rather then Tokyo's recent development into a mega-city.

A Brief history

The capital of Japan changed from Kyoto to Tokyo, then Edo, in 1868 to symbolize the transition of power from the traditional Shogun to the new Emperor Meji. With this shift in power Japan as whole began to rapidly industrialize with the help of foreign influence. One of the key symbols of industrialization at the time was having a developed transportation network, which Japan happened to lack. Emperor Meji ordered the construction of Japans new infrastructure and in 1870 the first Japanese railway was opened connecting Tokyo to Yokohama(Additional source).

From here Japan continued to rapidly expand its railway network to connect many of its already large cities with government support. Keep in mind that these railways were being built at a time when Japan had an authoritarian government and no existing infrastructure to compete with. Additionally the population of the area was significantly less then it was today. The greater Tokyo area at the time only had roughly 5 million people vs the over 40 million it homes today. Because of this development style and the current state of Japan at the time, Japans railroads actually became the foundation for urban development. As railroads went up around the Tokyo area and more people began to live there, more residential areas were constructed around the existing train infrastructure. Thus many of the lines that you see today in the Tokyo area and much of Japan have existed there since before the deelppment of the highly populated area that exists today. This explains why the Tokyo area today has many above ground lines in comparison to other large or mega cities.

In Modern Tokyo

In the Modern Tokyo area, post World War 2 to now, many of the existing train lines have remained in the places that they were since they were built in the late 19th and early 20th centuries. With weak eminent domain laws in Japan, the expansion of new railways in already existing residential zones is nearly impossible, but given that many important zones were developed around the railroads themselves this has rarely been viwed as necessary. In fact during its redevelopment phases following the war Japan wanted to alter some of its existing infrastructure networks, but lacked the means to do so. Instead Japan and the Tokyo area have pursued a policy of land readjustment to deal with some of the inefficiencies in the city. This lead to many state planned and sponsored railroads as the rail industry remained in the governments hands until the 1980's.

Additionally the train companies that are opening new lines, are heavily diversified companies that have divisions in real estate, commerce, banking, etc.. Giving them the ability to create new economic centers around their railroads thus incentiveizing citizens to use and live near existing or new railroads.

Conclusion

To summarize many of the Tokyo area trains were built prior to the areas development into a mega city. Instead the mega city was built around the existing railways. Many of the lines were built under an authoritarian government that had the power and resources to will into existences hundreds of thousands of miles of railways in Tokyo and across Japan. Today many of the new railroads that are opened in the Tokyo area are opened outside the major cities or very near existing lines without the need to clear existing buildings and operated by a multitude of private enterprises. Thus they have no need to compromise between the existing urban structures and the usage commuter rail lines. Furthermore Wikipedia says it best,

The development of Japan since 1872 is analogous to that of its rail network. Over this period, railways became the most important means of transport – especially for the movement of passengers – and they retain this role in the larger cities today. With many suburban cities having been developed by railway operators, the unchallenged importance of rail is something unique in the world.