How to combat self-doubt?

Here's a good guide by Ajahn Brahm on overcoming doubt. Doubt not just on the method, or the teachings, but also doubt on one's own abilities.

Please see the section on doubt from the essay "The Five Hindrances" by Ven. Ajahn Brahm:

Doubt refers to the disturbing inner questions at a time when one should be silently moving deeper. Doubt can question one's own ability "Can I do This?", or question the method "Is this the right way?", or even question the meaning "What is this?". It should be remembered that such questions are obstacles to meditation because they are asked at the wrong time and thus become an intrusion, obscuring one's clarity.

The Lord Buddha likened doubt to being lost in a desert, not recognising any landmarks.

Such doubt is overcome by gathering clear instructions, having a good map, so that one can recognise the subtle landmarks in the unfamiliar territory of deep meditation and so know which way to go. Doubt in one's ability is overcome by nurturing self confidence with a good teacher. A meditation teacher is like a coach who convinces the sports team that they can succeed. The Lord Buddha stated that one can, one will, reach Jhana and Enlightenment if one carefully and patiently follows the instructions. The only uncertainty is 'when'! Experience also overcomes doubt about one's ability and also doubt whether this is the right path. As one realised for oneself the beautiful stages of the path, one discovers that one is indeed capable of the very highest, and that this is the path that leads one there.

The doubt that takes the form of constant assessing "Is this Jhana?" "How am I going?", is overcome by realising that such questions are best left to the end, to the final couple of minutes of the meditation. A jury only makes its judgement at the end of the trial, when all the evidence has been presented. Similarly, a skilful meditator pursues a silent gathering of evidence, reviewing it only at the end to uncover its meaning.

The end of doubt, in meditation, is described by a mind which has full trust in the silence, and so doesn't interfere with any inner speech. Like having a good chauffeur, one sits silently on the journey out of trust in the driver.

I have been unsuccessful in worldly matters, I was unsuccessful in getting 'this and that', I couldn't fulfill the expectations I had...then how do you expect to succeed in attaining Nirvana...If you failed in samsaric life, then...so and so...

That line of thinking is more in-line with conceit than doubt. Doubt usually has a narrative that directly challenges the teachings, whereas conceit can take the form of various types of inferiority complexes, which I believe is the case for you. In any case, like Jade Empire says, one just notices conceit and its different levels - superiority - equality - inferiority - by tuning into the feeling-tone of the narrative - in your case it's unpleasant, so it is inferiority conceit.

The Upadanakkhandhas go like this...

The excitation of sankharas initiate thoughts - the mind detects those thoughts and uses the feeling/perception blueprint to determine if those thoughts are worthy of further entertainment. It considers them worthy and plays a repetitious rendition of those thoughts - this is the intention. This cycling leads to the gross feeling of unworthiness.

Don't try to reason with this part of the mind - it's neurotic. Reasoning with it means you've recognized its importance, and so it will show up again. You're creating this situation yourself, but have not yet developed your practice enough to see this at this depth. It will come to you.

That line of thinking would have me reflect, "This is stress; and this is the origin of stress; etc."

I think that is an essential benchmark (i.e. a metric or dimension against which to measure experience, and lines of thought).

A lot of that dialog seems to me based on risky assumptions: especially "I", "getting", "expect", and "success/failure".

The assumption seems faulty, i.e. that the "I" can reliably acquire because of who it is; or that attainment is about getting.

I think that the "I" itself -- i.e. notions about the "I" -- is some kind of fetter.

Perhaps that's why you can read about "confidence in the Dhamma" but not about "self-confidence".

I could quote suttas but perhaps you know them.

At one level, there is cause and effect -- but I think it's misleading to associate that with "I".

I remember that a preschool teacher is taught that she may observe, distinguish, and teach the difference between what she'd call "good and bad behaviour" -- but it's the behaviour that's good or bad, not the person, and she shouldn't teach, "You are a bad boy."

It's maybe more helpful to see things -- actions, intentions, thoughts, conditions -- without the view of their being specifically embodied in this or that person. And maybe it's help to see people (including yourself) as impermanent, inconstant. I mean the line of reasoning you're complaining about is, "Because I was unsuccessful in the past I will be unsuccessful in the future", is that meant to imply you've never ever learned to do anything, never adapted to changing circumstance, never learned a new skill?

But one of the "new skills" of Buddhism is perhaps to recognise stress when you observe it.

For only stress is what comes to be;
stress, what remains & falls away.
Nothing but stress comes to be.
Nothing ceases but stress.

SN 5.10

How does one counter this negative chatter and some kind of predictive reasoning?

Just be mindful of it - mindfulness will keep it from proliferating. This is just thinking and should be noted as "thinking, thinking". If any aversion towards the thoughts arises then the aversion should be taken as an object and noted as "disliking, disliking".

Observe and note it until it passes away - if it doesn't pass away after a long time just go back to the rising and falling of the abdomen. If it arises again the same technique should be used.

In short, turn the negative chatter/predictive reasoning into a meditation object, study and learn about it and cultivate Wisdom (Paññā) from it. Wisdom will eradicate doubt.