In what ways are the Advaita concept of 'Nirguna Brahman' and the Buddhist concept of Buddha Nature similar and different?

Generally speaking, Buddha Nature teachings are part of the Third Turning of the Wheel of Dharma and thus require the wisdom realizing emptiness described in the Second Turning of the Wheel of Dharma to interpret correctly. In this regard, I don't think they can be equated in any way with Vedanta. See here for more on how to regard the provisional and definitive meanings of the Three Turnings of the Wheel of Dharma.

I think what Buddhism teaches is entirely different in this context. I will resort to logic to answer some of your questions. Below I question the understanding of Nirguna Brahman:

Shankara asserts that each observer consciousness is identical with nirguna brahman (brahman without qualities), and that with liberation, the inner physical organ associated with that observer consciousness returns to maya-illusion, and that the observer consciousness merges with brahman without qualities.

If nirguna brahman is without qualities then how is it known ? It can not be known.

Secondly, if nirguna brahman is an eternal,permanent entity and everybody merges with it then why do karma ? One can kill himself right now and achieve the nirguna brahman. But that doesnt happen. Clearly your nirguna brahman (brahman without qualities) demands qualities in order to merge!! But being himself nirguna(without qualities) how can it demand saguna(good qualities) ?

Thirdly , observer consciousness which merges with the nirguna brahman can not be you or me. Buddha says self is not worth identifying as observer or the participant.In short Observer consciousness which gets liberated is not you or me or anyone because there is no self which is permanent and unchanging.

To answer your question

In what ways are the Advaita concept of 'Nirguna Brahman' similar, and/or different, to the Buddhist concept of Buddha Nature?

Nirguna Brahman and Nature of Tathagata can not be compared. Nirguna Brahman is said to exist. Tathagata is not said to be existing or non-existing.

Buddha Nature is known as "tathagatagarbha".

According to this article:

Tathagatagarbha, or Tathagata-garbha, means "womb" (garbha) of Buddha (Tathagata). This refers to a Mahayana Buddhist doctrine that Buddha Nature is within all beings. Because this is so, all beings may realize enlightenment. Tathagatagarbha often is described as a seed, embryo or potentiality within each individual to be developed.

And according to this article:

The 'tathagatagarbha' symbolizes the potential for enlightenment (a principle) rather than a material "essence" of ultimate truth.

Buddha Nature is said to originate from the concept of the "luminous mind" in Pabhassara Sutta, which is anyway not eternal, not unconditioned and not impermanent (see this answer).

Nirguna Brahman is described in Advaita as the substratum of all phenomena in Vivekachudamani 289 and as the material cause of the phenomenal universe in Aparokshanubhuti 45.

So, while Buddha Nature is simply the potential for enlightenment (a principle), it is not the material "essence" of ultimate truth, which Nirguna Brahman is. So Buddha Nature and Nirguna Brahman are not the same.

The second concept often wrongly equated with Nirguna Brahman, is Madhyamaka's "emptiness of emptiness" or shunyata. This answer in Hinduism.SE explains why both are not the same. Basically, Nirguna Brahman is empty of qualities but not empty of substance. Meanwhile, shunyata is empty of substance and even this emptiness itself is empty of substance or existence. In other words, Nirguna Brahman is a Transcendental Absolute Reality, while shunyata is devoid of any transcendental or absolute reality.

This is supported by Banaras Hindu University Professor T. R. V. Murti's statement (quoted below) in this book chapter: Murti T.R.V. (1973) Saṁvṛti and Paramārtha in Mādhyamika and Advaita Vedānta. In: Sprung M. (eds) The Problem of Two Truths in Buddhism and Vedānta. Springer, Dordrecht:

It has been the fashion to consider that the differences between the Madhyamika śūnyatā and Brahman are rather superficial and even verbal, and that the two systems of philosophy are almost identical. At least Professor Radhakrishnan thinks so, and Stcherbatsky's and Dasgupta's views are not very different. I hold a contrary view altogether: that in spite of superficial similarities in form and terminology, the differences between them are deep and pervasive.

The third concept usually mistaken for Brahman is the concept of the Eternal Buddha. This comes from the notion that the Buddha's dharmakaya or Dharma Body is Eternal. This does not mean that the Buddha is an eternal being or entity. Rather, it means that the Buddha's teachings (called the Dharma or Dhamma) is eternal. This originates from the Vakkali Sutta where the Buddha told Vakkali:

"Enough, Vakkali! What is there to see in this vile body? He who sees Dhamma, Vakkali, sees me; he who sees me sees Dhamma. Truly seeing Dhamma, one sees me; seeing me one sees Dhamma."

And also from the Mahaparinibbana Sutta:

Then the Blessed One said to Ven. Ananda, "Now, if it occurs to any of you — 'The teaching has lost its authority; we are without a Teacher' — do not view it in that way. Whatever Dhamma & Vinaya I have pointed out & formulated for you, that will be your Teacher when I am gone.

All Vedanta schools accept that Atman is eternal (according to Bhagavad Gita 2.20), and Brahman is absolute and eternal.

Meanwhile, the anatman or anatta teaching subscribed by all schools of Buddhism, states that there is no eternal self (i.e. no permanent standalone eternal entity) in all phenomena.

The eminent German indologist Professor Helmuth von Glasenapp explains aptly in his essay "Vedanta and Buddhism: A Comparative Study":

Nothing shows better the great distance that separates the Vedanta and the teachings of the Buddha, than the fact that the two principal concepts of Upanishadic wisdom, Atman and Brahman, do not appear anywhere in the Buddhist texts, with the clear and distinct meaning of a "primordial ground of the world, core of existence, ens realissimum (true substance)," or similarly.

For more discussion on the differences between doctrines in Hinduism and Buddhism, please see this answer.

With respect to the Advaita Vedanta, I assume your question relate to the idea that "All is one, one is all" which expresses non-duality (hence the tag 'nonduality'). From there, I assume you have in mind that the Advaita Vedanta posit this 'oneness' (or one primordial ground of which varieties are a manifestation) as something akin to Buddha Nature.

If it is what you have in mind, that is not Buddhist, and it is not what Buddhist have in mind when they speak of Buddha Nature. In addition, the Buddhist conception of non-duality is different.

From the Madhyamika perspective, for instance, there are three types of dualistic appearances:

1. The appearance of conventionalities
2. Appearance of inherent existence
3. Appearance of apprehending subject and apprehended object as being two different substances

Now, while Madhyamika posit that apprehended object and apprehending consciousness are different substances, Cittamatra assert that apprehended object and apprehending consciousness are the same substance. According to Cittamatra tenets, the apprehended object and the apprehending consciousness both arise simultaneously from a seed that lies in the mind-basis-of-all (alaya vijnana). The mind-basis-of-all being a type of consciousness (the 8th) that is neutral and acts as a repository, and is the object-possessor of the seeds that are "deposited" in it.

Even from this perspective, there isn't anything like a notion of "oneness" similar to that of some Hindu systems. Especially, even in the case where phenomena are not established externally (to consciousness), persons are different from each other and do not emanate from a common ground.

As to Buddha Nature, without going in the details, Karl Brunnholzl gives a decent list of what it is taken to be in different scriptures. It is in 'When the Clouds Part: The Uttaratantra and Its Meditative Tradition as a Bridge between Sutra and Tantra'.

Different Ways of Explaining the Meaning of Tathagatagarbha:

• Tathagatagarbha as emptiness
• Tathagatagarbha as Minds Luminous Nature
• Tathagatagarbha as the Alaya-Consciousness
• Tathagatagarbha as a Sentient Being
• Tathagatagarbha as the Dharmakàya, Suchness, the Disposition, and Nonconceptuality

One short paper that compares and contrasts Vedanta and Buddhism is Acharya Mahayogi Sridhar Rana Rinpoche's essay "Madhyamika Buddhism vis-à-vis Hindu Vedanta".

When we compare the Advaita Vedanta, especially as interpreted by Shankara and Madhyamika, whether be it the Svatantric from of Bhabya or Prasangic form of Candrakirti, the sharing of the same language, culture and analogies while talking about two different paradigms becomes obvious. Because of the use of the same language structure (be it Pali or Sanskrit) and the same analogies to express two different paradigms many Vedantins or scholars of Buddhism with Vedantic backgrounds have been fooled into thinking Buddhist Madhyamika is a re-interpretation of Hindu Vedanta.

In fact, the Buddha, after long years of Brahmanic as well as Sramanic meditation, found the concept of Brahma (an ultimately real, unchanging, eternal substratum to this ephemeral transient world) not only inadequate to solve the basic issue of humanity- i. e. sorrow (dukha) and questioned the very existence of such an eternal substratum; but also declared that a search for such an imagined (Skt. Parikalpita Atman) Brahman was a form of escapism and therefore not really spiritual [...]

[...]

Since the concept of Brahman, the truly existent (Skt. paramartha sat) is the very foundation of Hinduism (as a matter of fact some form of an eternal ultimate reality whether it is called God or Nature is the basis of all other religious systems); when Buddhism denies such an ultimate reality (Skt. paramartha satta) in any form, it cuts at the very jugular veins of Hinduism.

[...]

Many Hindu scholars believe that without an ultimate eternal reality then there can be no liberation from the changing, transient samsara; therefore even though the Buddha denied the ultimate reality, he could have meant only conceptually really existing reality, no the eternal ultimate reality which is beyond concepts. Otherwise there cannot be liberation. The fault with this kind of thinking is that it is measuring the thesis (which is no thesis) of the Buddha (or interpreting the Buddha) from within the Hindu paradigm. Remaining within the Hindu paradigm, an eternal ultimate reality is a necessity (a necessary dead end as the Buddha saw it) for the soteriological purpose, i. e. for liberation. Since according to the Buddha there is no Brahman- such a concept being merely an acquired fabrication (skt: parikalpana) learned from wrong (skt: mithya) scriptures, hankering after, searching for such a Brahman is necessary a dead end which leads nowhere, let alone liberation. The Buddhist paradigm if understood correctly, does not require an eternal something or other for liberation. In Buddhism liberation is not realizing such a ground but rather a letting go of all grounds, i. e. realizing groundless.

[...]

But even though Sankara copied the use of these words from Buddhism and also copied many other conceptual words from Nagarjuna to elucidate his Vedantic paradigm, the paradigm that he tries to clarify with these words different. [...] In the Vedantic context, the relative truth (Skt. samvritti satya) is that this samsara is an illusion and the ultimate truth (skt. paramartha satya) is that there is an ultimately existing thing (skt. paramartha satta) transcending/ immanent in this world. The relative truth will vanish like a mist and the both transcendent and immanent Brahman will appear as the only Truth, the world being false. To sum it up, the Vedantic ultimate truth is the existence of an ultimate existence or ultimate reality. Reality here is used as something which exists (skt. satta). However, the Buddhist ultimate truth is the absence of any such satta i. e. ultimately existing thing or ultimate reality.