What is the Buddhist text quoted by the Hindu philosopher Yogaraja?

The works of Yogarāja being very old clearly tells Anatta (not self) means not self of 'I' which is psycho-phyical. The term self in the 'not self' doesn't mean lack of super consciousness but lack of psycho-phyical consciousness. E.g, When we say - I'm doing, I'm eating, I'm playing etc. Here 'I' is psycho-phyical not real 'I'. Real 'I' is unchanging & permanent which is not supposed to do something, eat something, or play etc. This psycho-physical I is called as not-self because it is not our real self. That is what meant by Yogarāja, that I can say as I belong to the system you have mentioned Kashmiri Shaivism. So there is no conflict between Buddhists & Kashmiri Shaivism here. I've explained similar concept here. Also, read this one as well.

Following is an excerpt from Anatta-lakkhana Sutta: The Discourse on the Not-self Characteristic.
There he addressed the bhikkhus of the group of five: "Bhikkhus." — "Venerable sir," they replied. The Blessed One said this.

"Bhikkhus, form is not-self. Were form self, then this form would not lead to affliction, and one could have it of form: 'Let my form be thus, let my form be not thus.' And since form is not-self, so it leads to affliction, and none can have it of form: 'Let my form be thus, let my form be not thus.'

"Bhikkhus, feeling is not-self...

"Bhikkhus, perception is not-self...

"Bhikkhus, determinations are not-self...

"Bhikkhus, consciousness is not self. Were consciousness self, then this consciousness would not lead to affliction, and one could have it of consciousness: 'Let my consciousness be thus, let my consciousness be not thus.' And since consciousness is not-self, so it leads to affliction, and none can have it of consciousness: 'Let my consciousness be thus, let my consciousness be not thus.'

"Bhikkhus, how do you conceive it: is form permanent or impermanent?" — "Impermanent, venerable Sir." — "Now is what is impermanent painful or pleasant?" — "Painful, venerable Sir." — "Now is what is impermanent, what is painful since subject to change, fit to be regarded thus: 'This is mine, this is I, this is my self'"? — "No, venerable sir."

"Is feeling permanent or impermanent?...

"Is perception permanent or impermanent?...

"Are determinations permanent or impermanent?...

"Is consciousness permanent or impermanent?" — "Impermanent, venerable sir." — "Now is what is impermanent pleasant or painful?" — "Painful, venerable sir." — "Now is what is impermanent, what is painful since subject to change, fit to be regarded thus: 'This is mine, this is I, this is my self'"? — "No, venerable sir."

"So, bhikkhus any kind of form whatever, whether past, future or presently arisen, whether gross or subtle, whether in oneself or external, whether inferior or superior, whether far or near, must with right understanding how it is, be regarded thus: 'This is not mine, this is not I, this is not myself.'

"Any kind of feeling whatever...

"Any kind of perception whatever...

"Any kind of determination whatever...

"Any kind of consciousness whatever, whether past, future or presently arisen, whether gross or subtle, whether in oneself or external, whether inferior or superior, whether far or near must, with right understanding how it is, be regarded thus: 'This is not mine, this is not I, this is not my self.'

"Bhikkhus, when a noble follower who has heard (the truth) sees thus, he finds estrangement in form, he finds estrangement in feeling, he finds estrangement in perception, he finds estrangement in determinations, he finds estrangement in consciousness.

"When he finds estrangement, passion fades out. With the fading of passion, he is liberated. When liberated, there is knowledge that he is liberated. He understands: 'Birth is exhausted, the holy life has been lived out, what can be done is done, of this there is no more beyond.'"

That is what the Blessed One said. The bhikkhus were glad, and they approved his words.

Now during this utterance, the hearts of the bhikkhus of the group of five were liberated from taints through clinging no more.

Note - Consciousness meant by Gautam Buddha is psycho-phyical self explained above, not real self.

I can find a "reverse" message in Udana 5.1 to "When one looks upon himself as the Self, he regards himself as a unique being due to the distinction between the Self and another being [i.e. the not-self]; this causes bondage and hostility [between the Self and the not-self]. All evils ensue from the assumption of such bondage.":

When the king, descending from the palace, went to the Blessed One and, on arrival, having bowed down to him, sat to one side. As he was sitting there, he said to the Blessed One, "Just now, when I had gone with Queen Mallikā to the upper palace, I said to her, 'Mallikā, is there anyone dearer to you than yourself?'

"When this was said, she said to me, 'No, great king. There is no one dearer to me than myself. And what about you, great king? Is there anyone dearer to you than yourself?'

"When this was said, I said to her, 'No, Mallikā. There is no one dearer to me than myself.'"

Then, on realizing the significance of that, the Blessed One on that occasion exclaimed:

Searching all directions with your awareness, you find no one dearer than yourself. In the same way, others are thickly dear to themselves. So you shouldn't hurt others if you love yourself.

I'm not sure it is a Buddhist message; though it's hard to be sure, something may be added or lost when it's translated.

When one looks upon himself as the Self, he regards himself as a unique being due to the distinction between the Self and another being [i.e. the not-self]; this causes bondage and hostility [between the Self and the not-self]. All evils ensue from the assumption of such bondage.

I think that Buddhism talks about skandhas and, instead of "When one looks upon himself as the Self", it's more likely to say something like, "When one looks upon upon the body as the self ... When one looks upon upon the perceptions as the self ... etc."

Your quote implies that there is a "himself" to look upon, and a "Self" that he can mistake himself for. That quote, with a capitalized S for Self, seems to me more at home in Shivaite doctrine than Buddhist.

As well as from doctrine about the aggregates being "non-self" (anatta), Buddhism also has doctrine about "pride", "arrogance", or "conceit" ... maybe this is part of the doctrine, or the kind of doctrine, that the quote is referring to.

Also, non-hostility is core to Buddhist doctrine (see e.g. adosa and the brahmaviharas).

Finally, people's answers to your question are quoting from the Pali canon; but an "11th century Kashmiri Shaivite philosopher" might be quoting from a different school of Buddhism (perhaps from the beginnings of what became Tibetan Buddhism).