What are the boundaries of existence and non-existence in Buddhism?

The middle ground on which buddha rest is in paticca samuppada. which itself is in between existence and phenomena.

Most people wrongly interpret 'anatta' to sanskrit anatman. and this is historical mistake. but I never find any words uttered by buddha to deny existence or to be specific deny soul.

If you closely study buddhas teaching. Soul is referred every where. But its called dukkha.

Whole teaching of buddha is revolving around dukkha. And in fact dukkha means soul. Nothing else.

Ok if you skeptical. There is other name.its called 5 aggregates. And buddha frequently says 5 grasping aggregate is dukkha.

In other traditions(hindu/Christian/islam) soul is called source of knowledge. Thats exactly 5 grasping aggregates.. Or in simple word its memory that participate in building perception/saññā

Memory = soul = dukkha.

And soul exist until you are arhat/ buddha.

we all exist. we all stay connected to soul(5 grasping aggregates) until we reach arhat stage.

Only arhat is free from soul/dukkha/grasping-aggregates. so no more rebirth.

---- NOTE ----

Thanks all for all your concern. I know this seems bit away from buddhas teaching . but trust me its not. I request all not to , downvote this answer( let it be at -3) and let it be here as some food for thought for every one.

To my understanding, the Buddha discouraged such philosophical speculation on the nature, origin etc. of the physical world. He preferred to focus on the goal, which is the ending of suffering.

There the Buddha addressed the mendicants:

“Once upon a time, mendicants, a certain person left Rājagaha, thinking ‘I’ll speculate about the world.’ They went to the Sumāgadhā lotus pond and sat down on the bank speculating about the world. Then that person saw an army of four divisions enter a lotus stalk. When he saw this he thought, ‘I’ve gone mad, really, I’ve lost my mind! I’m seeing things that don’t exist in the world.’

Then that person entered the city and informed a large crowd, ‘I’ve gone mad, really, I’ve lost my mind! I’m seeing things that don’t exist in the world.’

‘But how is it that you’re mad? How have you lost your mind? And what have you seen that doesn’t exist in the world?’

‘Sirs, I left Rājagaha, thinking “I’ll speculate about the world.” I went to the Sumāgadhā lotus pond and sat down on the bank speculating about the world. Then I saw an army of four divisions enter a lotus stalk. That’s why I’m mad, that’s why I’ve lost my mind. And that’s what I’ve seen that doesn’t exist in the world.’

‘Well, mister, you’re definitely mad, you’ve definitely lost your mind. And you’re seeing things that don’t exist in the world.’

But what that person saw was in fact real, not unreal. Once upon a time, a battle was fought between the gods and the demons. In that battle the gods won and the demons lost. The defeated and terrified demons entered the citadel of the demons through the lotus stalk only to confuse the gods.

So mendicants, don’t speculate about the world. For example: the cosmos is eternal, or not eternal, or finite, or infinite; the soul and the body are the same thing, or they are different things; after death, a Realized One still exists, or no longer exists, or both still exists and no longer exists, or neither still exists nor no longer exists. Why is that? Because those thoughts aren’t beneficial or relevant to the fundamentals of the spiritual life. They don’t lead to disillusionment, dispassion, cessation, peace, insight, awakening, and extinguishment.

When you think something up, you should think: ‘This is suffering’ … ‘This is the origin of suffering’ … ‘This is the cessation of suffering’ … ‘This is the practice that leads to the cessation of suffering’. Why is that? Because those thoughts are beneficial and relevant to the fundamentals of the spiritual life. They lead to disillusionment, dispassion, cessation, peace, insight, awakening, and extinguishment.

That’s why you should practice meditation …”
SN 56.41

So, the questions are: Are fictional hobbits 'sat' or 'asat'?

I don't know what the Mahayana canon might say.

I think that according to the philosophy of modern science, a theory or statement is "true" if it is useful, a description of observations, and perhaps predictive -- including for example "Darwin's theory of evolution", "Newton's law of gravitation", "Einstein's relativity"; and "germ theory" in medicine, etc.

I find the Four Noble Truths to be excellent in this way -- useful and predictive and a good description of what I observe.

Following from this, I question, what is the "usefulness" of "a hobbit"? I think its use is to act as the narrative perspective:

Narrative perspective is the vantage point from which events of a story are filtered and then relayed to the audience.

The story begins with the Shire which is a place and society not very unlike England at the time, perhaps the reader would find it familiar and relate to that. The hobbits -- and therefore the reader -- gradually meet other characters and plot points, e.g. the wizard and other adventures.

So I guess that hobbits are true and useful within the context of the story.

The fact that that is only in a certain context (i.e. "within the story") isn't wholly bad -- e.g. Newton's laws too for example are only useful in a certain context, i.e. when you care to be solving that kind of problem, and even then not at "relativistic" speeds or "quantum" sizes.

Maybe whether it's useful (or true) isn't an inherent property of the thing.

Part of the genius of Buddhism is its formulation of a value system, i.e. that what's valuable is to reduce or end the suffering of sentient beings (and so to encourage good qualities and not bad).

Are hypotheticals 'sat' or 'asat'?

Perhaps "it depends" on how useful they are -- or how moral.

Consider "Russel's teapot" for example -- it's a hypothetical teapot but not IMO very useful, at best it's used to explain "falsifiability" which I find a dry subject (but I've never been taught philosophy).

Conversely consider a hypothetical crime or misdeed, or good deed -- and whether it would be for better or for worse if it were put into practice -- those may be a relatively useful hypotheticals, doctrine, guidelines, if it informs every-day morality and skilfulness. Or is that not so?

But maybe what's "true" in the latter case -- by analogy with the scientific "laws" or "theories" listed earlier -- is the moral law or law of consequences.

"If we allow for hobbits to exist, then we must also allow for omnipotent, self-created gods to exist, and souls."

I think this assertion is wrong in the context you assert in your addendum: conventional truth.

Your quote at the start of this answer is analogous to saying that if people mistakenly believe hobbits to exist conventionally, then they have to accept that souls truly exist or something. I don't think that is the case at all.

If someone mistakes a rope (conventionally existent thing) to be a snake on a moonlit night (conventionally non-existent thing eg a snake doesn't actually exist at all in that place where the rope is... much in the way a hobbit doesn't exist), that doesn't obligate them to accept that souls truly exist. The conclusion does not follow the premise.

UPDATE:

Your new claim still seems faulty:

“ If we allow for (fantasy) hobbits and if such rationale similarly pervades all non-existents, then we must allow for souls (because our reasons for how hobbits can exist do not exclude how souls can exist).”

I am guessing you wish to claim that a person believing that hobbits exist (from mistaken evidence… ie, they saw a small person with big feet and assumed they were looking at a hobbit) would be forced to conclude that souls exist. But this does not follow. Someone could mistakenly believe hobbits to exist, but still disbelieve that souls exist. The opposite is also possible: believing souls exist, but not hobbits.

In Pali, there is the word 'atthi', which means 'exists'. At times, 'atthi', such as in 'atthita' in SN 12.15, refers to a wrong view. At other times, 'atthi' refers to absolutely existing things, such as:

Atthi, bhikkhave, tadāyatanaṁ... Esevanto dukkhassā.

There exists, monks, that dimension... the end of suffering

Ud 8.1

“Well, is there no such thing as suffering?”

‘Kiṁ nu kho, bho gotama, natthi dukkhan’ti?

“It’s not that there’s no such thing as suffering.

‘Na kho, kassapa, natthi dukkhaṁ.

Suffering exists.”

Atthi kho, kassapa, dukkhan’ti.

SN 12.17

There's an issue here over the distinction between 'existence' and 'the knowledge of existence': i.e., objects vs mental objects. I mean, assume there's a cow standing in a field under the sky. Does the cow know that it's a cow, or that this is a field, or that that's the sky? Does the cow have mental objects? And if not, to what extent do these 'real' objects exist (as opposed to being, say, a singular cow-field-sky experiential thing)?

Hobbits, unicorns, turtle hair, etc., all exist as mental objects. As mental objects they are dependent arisings, and they are both causes and effects. They may not have a 'real' reference (whatever you decide that means), but they are talked about, speculated on, argued over, worked into feature films, and all that. Doubtless there are people for whom hobbits are more 'real' than actual living people from other countries...

The work here isn't to figure out which mental objects are 'real' and which aren't. The work is to recognize that no mental objects are 'real', and that we should be compassionate and circumspect about our attachments to them.

it is not a matter of boundaries, but a question of true or false. sabbe sankhara anicca means all conditioned things are transient, everchanging, impermanent, so from an ontological point of view illusory or non-existent. so only Nibbana, the Unconditioned, Asankhata is constant, always the same and eternal, meaning ontologically real and existent.

By & large, Kaccayana, this world is based upon a duality: existence and non-existence.

...

'Everything exists': Kaccayana, this is one extreme. ‘Nothing exists’: this is the second extreme. Kaccayana, without approaching either of these two extremes, the Tathāgata teaches Dhamma by the middle.

(SN 12.15)

The way I understand this is, the categories such as existence and non-existence are abstractions, and as such they try to describe and categorize the world in broad terms.

As with all abstractions though, when you zoom in, you see a bunch of extra details that don't fit simplistic categorization.

So when it comes to hobbits and other fantasies, and even natural phenomena such as e.g. "the horizon" or "a rainbow" - they satisfy some definitions of existence but not others — illusions exist as phenomena but "not in the way they appear" (a fairly standard Mahayana assertion, see e.g. Vasubandhu's commentary on Madhyantavibhaga), hobbits exist as representations in our minds while we're discussing them, and so on. This is to say that existence and non-existence are leaking abstractions. As with all abstractions, this may be a useful analytical tool when used appropriately, or it may be a source of dukkha if we get too hung up on it.

I am familiar with a standard Mahayana line of reasoning that differentiates between types of non-existence such as e.g. non-substantial-existence of rainbows and selves vs. "stronger" non-existence of turtle hair scarfs. The point is to highlight the difference between Emptiness and mere absence.

Then there is the "pragmatic" camp of Karma Kagyu that says, in the words of late Thrangu Rinpoche 🙏, that it's not as important whether this chair really exists or not, but our attitude towards the chair. Are we attached to this chair? Do we hate this chair? Do we think we are the chair? Are we free from this chair? That's what really matters in practice, not whether it exists or not - they say in Karma Kagyu, and I don't disagree.

And yet for me it goes still deeper than that. In my view, the arguments about semantics and hair-splitting definitions of various types of existence and non-existence of selves, rainbows, hobbits etc., make this a perfect case-study of Emptiness of phenomena at large. The issue is not with exhaustively identifying all types of existence and non-existence and giving them precise definitions. The issue is with phenomena themselves. They are not delineated ontologically, their delineation is part of the process of perception. Reality is an interpretation we make and delineating phenomena is a part of interpretation. Existence and non-existence (and everything in-between) are secondary to delineation.

When we say phenomena are empty (of svabhava aka intrinsic existence), we mean phenomena are perspective-dependent. When we clearly see that phenomena (including the mental phenomena such as the concepts!) are perspective-dependent, how can we seriously argue about existence and non-existence!