Do Buddhists claim that if there was a self it could not be annihilated?

No, they don't.

Those who claim a 'self' defined what they mean by the 'self' and some thinker at the time of Buddha, for example, the Sankhya philosophers posit a self that can not be annihilated and the Buddist show that it is not true.

It's like if group A says a unicorn exists. The Buddhist didn't say 'for a unicorn to exist it has to have a single horn on its head' it's not the business of the Buddist defining what that other group thinks, it's group A who claims to know a horse with a single spiral horn on its head.

So in short when the Buddhists say 'not-self...not-self' they are not predicting the existence of a thing called self.

If you ask them what is self they will probably say an idea by some stupid people, but they won't define it for you.

Perhaps this is what you are remembering:

The Fruits of the Ascetic Life

3.3. The Doctrine of Ajita Kesakambala One time, sir, I approached Ajita Kesakambala and exchanged greetings with him. When the greetings and polite conversation were over, I sat down to one side, and asked him the same question.

He said: ‘Great king, there is no meaning in giving, sacrifice, or offerings. There’s no fruit or result of good and bad deeds. There’s no afterlife. There’s no obligation to mother and father. No beings are reborn spontaneously. And there’s no ascetic or brahmin who is well attained and practiced, and who describes the afterlife after realizing it with their own insight. This person is made up of the four primary elements. When they die, the earth in their body merges and coalesces with the main mass of earth. The water in their body merges and coalesces with the main mass of water. The fire in their body merges and coalesces with the main mass of fire. The air in their body merges and coalesces with the main mass of air. The faculties are transferred to space. Four men with a bier carry away the corpse. Their footprints show the way to the cemetery. The bones become bleached. Offerings dedicated to the gods end in ashes. Giving is a doctrine of morons. When anyone affirms a positive teaching it’s just hollow, false nonsense. Both the foolish and the astute are annihilated and destroyed when their body breaks up, and don’t exist after death.’

And so, when I asked Ajita Kesakambala about the fruits of the ascetic life apparent in the present life, he answered with the doctrine of annihilationism. It was like someone who, when asked about a mango, answered with a breadfruit, or when asked about a breadfruit, answered with a mango. I thought: ‘How could one such as I presume to rebuke an ascetic or brahmin living in my realm?’ So I neither approved nor dismissed that statement of Ajita Kesakambala. I was displeased, but did not express my displeasure. Neither accepting what he said nor contradicting it, I got up from my seat and left.

A sutta possibly providing an answer is the 2nd and most important sermon of the Buddha, namely, the Anatta-Lakkhana Sutta: The Discourse on the Not-self Characteristic. Relevant paragraphs for consideration are:

"...if form were self, then form would not lead to affliction (ābādhāya) and it should obtain regarding form: 'May my form be thus, may my form not be thus'; and indeed, O monks, since form is not-self, therefore form leads to affliction and it does not obtain regarding form: 'May my form be thus, may my form not be thus.'

"... if feeling were self, then feeling would not lead to affliction and it should obtain regarding feeling: 'May my feeling be thus, may my feeling not be thus'; and indeed, O monks, since feeling is not-self, therefore feeling leads to affliction and it does not obtain regarding feeling: 'May my feeling be thus, may my feeling not be thus.'

"... if perception were self, then perception would not lead to affliction and it should obtain regarding perception: 'May my perception be thus, may my perception not be thus'; and indeed, O monks, since perception is not-self, therefore, perception leads to affliction and it does not obtain regarding perception: 'May my perception be thus, may my perception not be thus.'

"... if mental formations were self, then mental formations would not lead to affliction and it should obtain regarding mental formations: 'May my perception be thus, may my mental formations not be thus'; and indeed, O monks, since mental formations are not-self, therefore, mental formations lead to affliction and it does not obtain regarding mental formations: 'May my mental formations be thus, may my mental formations not be thus.'

"... if consciousness were self, then consciousness would not lead to affliction and it should obtain regarding consciousness: 'May my consciousness be thus, may my consciousness not be thus'; and indeed, O monks, since consciousness is not-self, therefore, consciousness leads to affliction and it does not obtain regarding consciousness: 'May my consciousness be thus, may my consciousness not be thus.'

While the above is referring to ābādhāya (sickness), it is indirectly saying, if there was a self, such a self should be permanent or not capable of sickness (and thus decay and death).

In addition to the answer by ChrisW, the following sutta quotes provide further perspective on the same topic.

From SN 22.93:

At Savatthi. “Bhikkhus, suppose there was a mountain river sweeping downwards, flowing into the distance with a swift current. If on either bank of the river kasa grass or kusa grass were to grow, it would overhang it; if rushes, reeds, or trees were to grow, they would overhang it. If a man being carried along by the current should grasp the kasa grass, it would break off and he would thereby meet with calamity and disaster; if he should grasp the kusa grass, it would break off and he would thereby meet with calamity and disaster; if he should grasp the rushes, reeds, or trees, they would break off and he would thereby meet with calamity and disaster.

“So too, bhikkhus, the uninstructed worldling … regards form as self, or self as possessing form, or form as in self, or self as in form. That form of his disintegrates and he thereby meets with calamity and disaster. He regards feeling as self … perception as self … volitional formations as self … consciousness as self, or self as possessing consciousness, or consciousness as in self, or self as in consciousness. That consciousness of his disintegrates and he thereby meets with calamity and disaster.

“What do you think, bhikkhus, is form permanent or impermanent?”—“Impermanent, venerable sir.”…—“Therefore … Seeing thus … He understands: ‘… there is no more for this state of being.’”

From MN 22:

“Venerable sir, can there be agitation about what is non-existent internally?”

“There can be, bhikkhu,” the Blessed One said. “Here, bhikkhu, someone has the view: ‘That which is the self is the world; after death I shall be permanent, everlasting, eternal, not subject to change; I shall endure as long as eternity.’ He hears the Tathāgata or a disciple of the Tathāgata teaching the Dhamma for the elimination of all standpoints, decisions, obsessions, adherences, and underlying tendencies, for the stilling of all formations, for the relinquishing of all attachments, for the destruction of craving, for dispassion, for cessation, for Nibbāna. He thinks thus: ‘So I shall be annihilated! So I shall perish! So I shall be no more!’ Then he sorrows, grieves, and laments, he weeps beating his breast and becomes distraught. That is how there is agitation about what is non-existent internally.”

“Venerable sir, can there be no agitation about what is non-existent internally?”

“There can be, bhikkhu,” the Blessed One said. “Here, bhikkhu, someone does not have the view: ‘That which is the self is the world…I shall endure as long as eternity.’ He hears the Tathāgata or a disciple of the Tathāgata teaching the Dhamma for the elimination of all standpoints, decisions, obsessions, adherences, and underlying tendencies, for the stilling of all formations, for the relinquishing of all attachments, for the destruction of craving, for dispassion, for cessation, for Nibbāna. He does not think thus: ‘So I shall be annihilated! So I shall perish! So I shall be no more!’ Then he does not sorrow, grieve, and lament, he does not weep beating his breast and become distraught. That is how there is no agitation about what is non-existent internally.

The first and perhaps most famous sutta like that which comes to mind is the Anatta-lakkhana Sutta: The Discourse on the Not-self Characteristic (SN22.59) translated here and here.

I think that says that if something (e.g. form or consciousness or feeling or perception) is impermanent, then it's "not fit" to be regarded as "self".

The word translated as "not fit" is kalla ("clever", "healthy", etc.).