Relying upon a Spiritual Guide, who created this idea? Is it actually a necessity for liberation?

Fire is a destructive force that will consume whatever we give it, yet in return, fire cooks our food, heats our homes, gives us light, and accompanies our reveries. Meditation is a kind of destructive force as well. Meditation helps us break the hold that our dramatic lives have on our emotions and thoughts. It also shows us the illusory nature of so much that we take to be real, that harms us, and it frees us from that suffering. Meditation breaks down the barriers that keep us constrained to just a limited corner of our otherwise powerful minds. And it allows us to find our way to our true nature. Finding this way is up to us, and that is what your quotation was saying.

However, like fire, mediation can have some powerful, and possibly harmful effects depending on how you react to them. That’s why it’s been said, in regard to meditation: It’s better not to start; but once started it’s better to finish. The ‘lifeguard’ that saves you from possibly being psychologically damaged by what you experience is a qualified teacher who knows what the expected experiences are that will arise, if you maintain a dedicated practice. And by “know” I don’t mean “read about it.” A qualified teacher will have been through it themselves, and will have internalized an explanatory framework to allow them to communicate what is happening to you, and why it’s happening. This is why you need a teacher.

Modern meditation that is offered up online and in classes are only meant to help you reduce stress, lose weight, get over emotional issues, and get the partner of your dreams (and whatever else the marketing department comes up with). The teachers often only know that much, and in many cases, haven’t gone beyond that point, so if you go too far with your mediation practice, you’re playing with fire, without a resource to protect you, because you don’t have a qualified teacher.

This, by the way, has inspired a whole new field of research that falls under the title: “The Dangers of Meditation.” Do a web search, there’s lots of articles. What it all boils down to though, is that modern secular meditation has been disassociated from the traditional explanatory frameworks that protect meditators. Outside of those frameworks, and Buddhism is (the best 😁) one, you fall into the fire analogy again—which in itself is proof that mediation has real effects, that go way beyond simple mindfulness.

Unfortunately, those that research these issues that arise from secular meditation practices, aren’t much more informed about traditional teaching of meditation, so they don’t realize the cause (that of having disassociated meditation from traditional explanatory frameworks) and just claim meditation is the cause, period, when (IMHO) it’s incompetence and ignorance, coupled with a large amount of hubris on the part of those who spawned the mindfulness industry without truly understanding what was at stake.

I added that last paragraph, to cutoff the possibility that someone reading this might get scared to meditate. Find a qualified teacher, listen to what they say, get their phone number, and then it’s all on you to practice.

Good luck!

In addition to the quote from this answer I'm reminded of advice from another (non-Buddhist) tradition: "Don't just study the lesson -- study the teacher".

I think that makes sense in any context -- if you really wanted to learn to practice Mathematics for example, you probably really need to study mathematicians and how they practice, and not simply study mathematics.

There's also a limit to what can be conveniently explained in words. In a Tai Chi class yesterday, the Master warned that some people read a lot and get the wrong the idea from books (instead they ought to practice a lot), and that sometimes books are misleading, and that you learn more from watching and copying the teacher. Incidentally he also said that when the teacher corrects you, then that's one thing; but pay attention when they correct someone else, because it's often easier to see someone else's mistake being corrected than to see it is to see your own mistake.

Also, your quote from the mahaparinibbana sutta: that was to his own students, after he had been teaching for 50 years or so, presumably to solidify their resolve and prepare them for his passing. What I said above (about needing a Master) may apply less when you are a Master yourself and no longer a novice -- when you "have had" a Master.

The term "spiritual guide" in the OP might perhaps be analogous to Kalyāṇa-mittatā in Pali and in the suttas.

This question is a perfect example of why you need a teacher.

Are we Mindful in Jhana?

The Sutras are a great 10,000 foot guide to Buddhism, but they lack specificity. It's akin to someone directing you to San Francisco to NY. The suttas pretty much say "go east young man". At best, they might tell you to pass through Chicago and Denver on the way. Your teacher (and commentarial literature) is responsible for fleshing out those directions. He might tell you to take interstate 80, avoid driving directly through Chicago, and to be sure to stop at some awesome truck stop on the way. He might even make sure you know how to drive a frigging car before sending you on your way.

The suttas are great, don't get me wrong, but you are going to get lost if you follow those directions alone. Further more, it's always nice to be able to stop for clarification along the way

In short it is one's own responsibility to make effort and finding suitable teachers is essentially all of that 'making an island'.

Canki Sutta explains how to choose a teacher who claims to have attained some distinct knowledge & vision.

Ariyapariyesana Sutta tells us how The Bodhisatta approached several teachers claiming various knowledge & vision.

Samadhi Sutta explains how a person who has attained either insight or tranquility should be looking to approach a suitable individual who can complement his development so that he can attain both tranquility & insight.

Ven. Ananda said to the Blessed One, "This is half of the holy life, lord: admirable friendship, admirable companionship, admirable camaraderie."

"Don't say that, Ananda. Don't say that. Admirable friendship, admirable companionship, admirable camaraderie is actually the whole of the holy life. When a monk has admirable people as friends, companions, & comrades, he can be expected to develop & pursue the noble eightfold path. - SN 45.2

I want to emphasise 'admirable' once again and that a teacher has to be carefully observed in regards to qualities of greed, anger & delusion before one visits & lends ear.

Relying upon a spiritual guide is a technique that helps one to develop faith in something other than your own troubled faculties. In my current understanding, this too can become troublesome. It can cause a fracture in the psyche, where the person always refers to an imagined entity for direction - which is another figment of the mind. It seems to be largely a Tibetan thing, but knowing how mastered the Tibetans are in this region, it probably works well for them and their students.

In all its rudiments, I think under well supervised conditions, what one is actually being taught (or should be taught) is how to reach that part of your mind that is receptive to the spiritual voyage. It is just that we often feel the need to personify that part of us, when in the clear light of day the spiritual guide grows into a sensitive, flexible, willing and responsive participant of this world which amounts to the simplicity of you, and not any ideas about other entities.

Is a teacher necessary for liberation and to what degree?

A teacher is necessary insofar as you cannot attain it on your own.

If you are so talented that you completely get it just from reading the Heart Sutra, or the Cuckoo's Song of Total Presence or the original suttas etc. then why would you need a guide. There are people like that, they are rare but they certainly do exist. There's nothing wrong with being a genius.

However if after reading some texts you feel like you still have a way to go, then it is only natural to rely on someone for guidance, at least to some degree. I mean, you asking this question here is an act of asking for guidance, isn't it.

And if you are really desperate for Enlightenment, it makes perfect sense to throw everything you've got at the teacher's feet and beg for instruction. The famous masters of the past did precisely this.

It is foolish to assume there were no people born before you who made it the task of their lives to get to the bottom of this Buddhism thing. If such people existed (and there were plenty of them in the last 2600+ years) it only makes sense to assume that some of them actually got it.

Now, if they got it, wouldn't it make sense for you to listen to what they have to say, to save you the hassle of stumbling on your own? Or is that too disturbing for the good old EGO, to have to follow a lead?

Finally, as the traditional metaphor goes: all rivers of India have their water come from the Himalayas. All instructions you get from the secondary texts and contemporary teachers are ultimately the Buddha's teaching. Relying on a good Buddha's student you're relying on the Buddha.

It's true that DN 16 says rely on yourself but it doesn't mean you don't need a teacher, just as a student in university relies on his own effort to study for exams, do assignments, projects etc. but may need guidance from his lecturer or professor.

The following sutta states that one who doesn't know what is to be known, should seek a teacher.

The term for teacher here is satthu in Pali. In Pali, there are no definite articles like "a" or "the" and also no upper case and lower case, as in "a teacher" and "the Teacher", so it may very well refer to the Teacher i.e. the Buddha, as it is found in Ven. Sujato's translation.

“Bhikkhus, one who does not know and see as it really is aging-and-death, its origin, its cessation, and the way leading to its cessation, should search for a teacher in order to know this as it really is.

“Bhikkhus, one who does not know and see as it really is birth … existence … clinging … craving … feeling … contact … the six sense bases … name-and-form … consciousness … volitional formations, their origin, their cessation, and the way leading to their cessation, should search for a teacher in order to know this as it really is.”
SN 12.82 (translated by Ven. Bodhi)

The following sutta shows five ways of learning and practice that leads to liberation. Only the first involves a teacher. So, a teacher is not absolutely necessary.

(1) “Here, bhikkhus, the Teacher or a fellow monk in the position of a teacher teaches the Dhamma to a bhikkhu. In whatever way the Teacher or that fellow monk in the position of a teacher teaches the Dhamma to the bhikkhu, in just that way he experiences inspiration in the meaning and inspiration in the Dhamma. As he does so, joy arises in him. When he is joyful, rapture arises. For one with a rapturous mind, the body becomes tranquil. One tranquil in body feels pleasure. For one feeling pleasure, the mind becomes concentrated. This is the first basis of liberation, by means of which, if a bhikkhu dwells heedful, ardent, and resolute, his unliberated mind is liberated, his undestroyed taints are utterly destroyed, and he reaches the as-yet-unreached unsurpassed security from bondage.

(2) “Again, neither the Teacher nor a fellow monk in the position of a teacher teaches the Dhamma to a bhikkhu, but he himself teaches the Dhamma to others in detail as he has heard it and learned it. In whatever way the bhikkhu teaches the Dhamma to others in detail as he has heard it and learned it, in just that way, in relation to that Dhamma, he experiences inspiration in the meaning and inspiration in the Dhamma. As he does so, joy arises in him. When he is joyful, rapture arises. For one with a rapturous mind, the body becomes tranquil. One tranquil in body feels pleasure. For one feeling pleasure, the mind becomes concentrated. This is the second basis of liberation, by means of which, if a bhikkhu dwells heedful, ardent, and resolute, his unliberated mind is liberated, his undestroyed taints are utterly destroyed, and he reaches the as-yet-unreached unsurpassed security from bondage.

(3) “Again, neither the Teacher nor a fellow monk in the position of a teacher teaches the Dhamma to a bhikkhu, nor does he himself teach the Dhamma to others in detail as he has heard it and learned it, but he recites the Dhamma in detail as he has heard it and learned it. In whatever way the bhikkhu recites the Dhamma in detail as he has heard it and learned it, in just that way, in relation to that Dhamma, he experiences inspiration in the meaning and inspiration in the Dhamma. As he does so, joy arises in him. When he is joyful, rapture arises. For one with a rapturous mind, the body becomes tranquil. One tranquil in body feels pleasure. For one feeling pleasure, the mind becomes concentrated. This is the third basis of liberation, by means of which, if a bhikkhu dwells heedful, ardent, and resolute, his unliberated mind is liberated, his undestroyed taints are utterly destroyed, and he reaches the as-yet-unreached unsurpassed security from bondage.

(4) “Again, neither the Teacher nor a fellow monk in the position of a teacher teaches the Dhamma to a bhikkhu, nor does he teach the Dhamma to others in detail as he has heard it and learned it, nor does he recite the Dhamma in detail as he has heard it and learned it, but he ponders, examines, and mentally inspects the Dhamma as he has heard it and learned it. In whatever way the bhikkhu ponders, examines, and mentally inspects the Dhamma as he has heard it and learned it, in just that way, in relation to that Dhamma, he experiences inspiration in the meaning and inspiration in the Dhamma. As he does so, joy arises in him. When he is joyful, rapture arises. For one with a rapturous mind, the body becomes tranquil. One tranquil in body feels pleasure. For one feeling pleasure, the mind becomes concentrated. This is the fourth basis of liberation, by means of which, if a bhikkhu dwells heedful, ardent, and resolute, his unliberated mind is liberated, his undestroyed taints are utterly destroyed, and he reaches the as-yet-unreached unsurpassed security from bondage.

(5) “Again, neither the Teacher nor a fellow monk in the position of a teacher teaches the Dhamma to a bhikkhu, nor does he teach the Dhamma to others in detail as he has heard it and learned it, nor does he recite the Dhamma in detail as he has heard it and learned it, nor does he ponder, examine, and mentally inspect the Dhamma as he has heard it and learned it, but he has grasped well a certain object of concentration, attended to it well, sustained it well, and penetrated it well with wisdom. In whatever way the bhikkhu has grasped well a certain object of concentration, attended to it well, sustained it well, and penetrated it well with wisdom, in just that way, in relation to that Dhamma, he experiences inspiration in the meaning and inspiration in the Dhamma. As he does so, joy arises in him. When he is joyful, rapture arises. For one with a rapturous mind, the body becomes tranquil. One tranquil in body feels pleasure. For one feeling pleasure, the mind becomes concentrated. This is the fifth basis of liberation, by means of which, if a bhikkhu dwells heedful, ardent, and resolute, his unliberated mind is liberated, his undestroyed taints are utterly destroyed, and he reaches the as-yet-unreached unsurpassed security from bondage.
AN 5.26

The following Vinaya quote shows that the Buddha intended for monks-in-training to definitely have a preceptor. That's absolutely required.

At that time the monks did not have preceptors or teachers, and as a result they were not being instructed. While walking for almsfood, they were shabbily dressed and improper in appearance. While people were eating, they held out their almsbowls to receive leftovers, even right over their food, whether it was staple food, non-staple food, delicacies, or drinks. They ate bean curry and rice that they themselves had asked for, and they were noisy in the dining hall. People complained and criticized them, “How can the Sakyan ascetics act like this? They are just like brahmins at a brahminical meal.”

The monks heard the complaints of those people. The monks of few desires who had a sense of conscience, who were contented, afraid of wrongdoing, and fond of the training, complained and criticized them, “How can monks act like this?”

They then told the Buddha.

Soon afterwards the Buddha had the Sangha gathered and questioned the monks: “Is it true, monks, that monks act like this?”

“It’s true, Sir.”

The Buddha criticzed them, “It’s not suitable for those foolish men, it’s not proper, it’s not worthy of an ascetic, it’s not allowable, it’s not to be done. How can they act like this? This won’t give rise to confidence in those without it or increase the confidence of those who have it, but it will hinder confidence in those without it, and it will cause some with confidence to change their minds.”

Then the Buddha spoke in many ways in dispraise of being difficult to support and maintain, in dispraise of great desires, discontent, socializing, and laziness; but he spoke in many ways in praise of being easy to support and maintain, of fewness of wishes, contentment, self-elimination, ascetic practices, serenity, reduction in things, and of being energetic. After giving a teaching on what is right and proper, he addressed the monks:

“There should be a preceptor.

The preceptor should regard his student as a son and the student his preceptor as a father. In this way they will respect, esteem, and be considerate toward each other, and they will grow and reach greatness on this spiritual path.

And a preceptor should be chosen like this:

After putting his upper robe over one shoulder, a student should pay respect at the feet of a potential preceptor. He should then squat on his heels, raise his joined palms, and say, ‘Venerable, please be my preceptor.’ And he should repeat this a second and a third time. If the other makes it understood by body, by speech, or by body and speech: ‘Yes,’ or, ‘No problem,’ or, ‘It’s suitable,’ or, ‘It’s appropriate,’ or, ‘Carry on with inspiration’—then a preceptor has been chosen. If the other doesn’t make it understood by body, by speech, or by body and speech, then a preceptor hasn’t been chosen.
Pli Td Kv1 sub-chapter 15

The qualities of a good teacher and preceptor can be found in Pli Td Kv1 sub-chapter 16, Mil 4.5, AN 5.159.

And finally we have the famous Rhinoceros Sutta telling us that for renunciants on the path to liberation, it's best to find companions who are on par or better. If not found, then go alone.

If you gain a mature companion,
a fellow traveler, right-living & wise,
overcoming all dangers
go with him, gratified,
mindful.

If you don't gain a mature companion,
a fellow traveler, right-living & wise,
wander alone
like a king renouncing his kingdom,
like the elephant in the Matanga wilds,
his herd.

We praise companionship — yes!
Those on a par, or better,
should be chosen as friends.
If they're not to be found,
living faultlessly,
wander alone
like a rhinoceros.
Rhinoceros Sutta (Snp 1.03)