Why should I focus on that which is not mine?

Be careful of succumbing yourself to a non-right view.

Bhikkhu Bodhi answered this very question in "The Balanced Way":

Like a bird in flight borne by its two wings, the practice of Dhamma is sustained by two contrasting qualities whose balanced development is essential to straight and steady progress. These two qualities are renunciation and compassion. As a doctrine of renunciation the Dhamma points out that the path to liberation is a personal course of training that centers on the gradual control and mastery of desire, the root cause of suffering. As a teaching of compassion the Dhamma bids us to avoid harming others, to act for their welfare, and to help realize the Buddha's own great resolve to offer the world the way to the Deathless.

Considered in isolation, renunciation and compassion have inverse logics that at times seem to point us in opposite directions. The one steers us to greater solitude aimed at personal purification, the other to increased involvement with others issuing in beneficent action. Yet, despite their differences, renunciation and compassion nurture each other in dynamic interplay throughout the practice of the path, from its elementary steps of moral discipline to its culmination in liberating wisdom. The synthesis of the two, their balanced fusion, is expressed most perfectly in the figure of the Fully Enlightened One, who is at once the embodiment of complete renunciation and of all-embracing compassion.

Both renunciation and compassion share a common root in the encounter with suffering. The one represents our response to suffering confronted in our own individual experience, the other our response to suffering witnessed in the lives of others. Our spontaneous reactions, however, are only the seeds of these higher qualities, not their substance. To acquire the capacity to sustain our practice of Dhamma, renunciation and compassion must be methodically cultivated, and this requires an ongoing process of reflection which transmutes our initial stirrings into full-fledged spiritual virtues.

The framework within which this reflection is to be exercised is the teaching of the Four Noble Truths, which thus provides the common doctrinal matrix for both renunciation and compassion. Renunciation develops out of our innate urge to avoid suffering and pain. But whereas this urge, prior to reflection, leads to an anxious withdrawal from particular situations perceived as personally threatening, reflection reveals the basic danger to lie in our existential situation itself — in being bound by ignorance and craving to a world which is inherently fearsome, deceptive and unreliable. Thence the governing motive behind the act of renunciation is the longing for spiritual freedom, coupled with the recognition that self-purification is an inward task most easily accomplished when we distance ourselves from the outer circumstances that nourish our unwholesome tendencies.

Compassion develops out of our spontaneous feelings of sympathy with others. However, as a spiritual virtue compassion cannot be equated with a sentimental effusion of emotion, nor does it necessarily imply a dictum to lose oneself in altruistic activity. Though compassion surely includes emotional empathy and often does express itself in action, it comes to full maturity only when guided by wisdom and tempered by detachment. Wisdom enables us to see beyond the adventitious misfortunes with which living beings may be temporarily afflicted to the deep and hidden dimensions of suffering inseparable from conditioned existence. As a profound and comprehensive understanding of the Four Noble Truths, wisdom discloses to us the wide range, diverse gradations, and subtle roots of the suffering to which our fellow beings are enmeshed, as well as the means to lead them to irreversible release from suffering. Thence the directives of spontaneous sympathy and mature compassion are often contradictory, and only the latter are fully trustworthy as guides to beneficent action effective in the highest degree. Though often the judicious exercise of compassion will require us to act or speak up, sometimes it may well enjoin us to retreat into silence and solitude as the course most conducive to the long-range good of others as well as of ourselves.

In our attempt to follow the Dhamma, one or the other of these twin cardinal virtues will have to be given prominence, depending on our temperament and circumstances. However, for monk and householder alike, success in developing the path requires that both receive due attention and that deficiencies in either gradually be remedied. Over time we will find that the two, though tending in different directions, eventually are mutually reinforcing. Compassion impels us toward greater renunciation, as we see how our own greed and attachment make us a danger to others. And renunciation impels us toward greater compassion, since the relinquishing of craving enables us to exchange the narrow perspectives of the ego for the wider perspectives of a mind of boundless sympathy. Held together in this mutually strengthening tension, renunciation and compassion contribute to the wholesome balance of the Buddhist path and to the completeness of its final fruit.

There is a Buddhist concept of Kalyāṇa-mittatā, Buddha definitely approved friendship amongst the sangha as well as for householders. As long as it was with right people (meaning friends addicted to gambling, alcohol should be avoided).

Now to your dilemma,

when I realised none of them are mine

The better thing to realize was to say, 'I am not' 'anatta'. As long as you have not realized no-self, and as long as you are you can say 'none of them are mine'. Of course, you do not possess them, so in that sense, they are not yours, but they are YOUR friends. As a householder, friendship is imperative, you cannot live in this samsara like an island, you never know when will you need whom. Although this should not be the criteria to make friends, but this is true.

They will change or perish. I felt detached and disconnected. I no longer focused on what they were saying.

They will change and perish so will you, so will everybody you know, your wife, your children, your parents, everything is impermanent, 'annicca'. The feeling of detachment is alright, in fact good, but dis-connection shows you lack respect and compassion. Everyone wants to tell their story to friends, share the problems and good and bad times with friends, you should have compassion. You should have listened to them without attachment. Like a lotus in a pond, be in this world but not of it.

I appeared absent-minded to them.

Are you not practicing mindfulness? Absent-mindedness is not something for a meditator. You should be aware. Later on, you will regret if only I would have been listening to him...

I appeared as one who doesn't understand or care or love.

Cultivate kindness and compassion. Metta bhava is important.

Why should I focus on people or serve people who are not mine?

Because Samsara is Nirvana (this is something I learned in Zen). Also not focusing on people who are not of profit to you or not yours shows selfishness and will bolster your ego. Remember the Jataka tales and remember all the sacrifice the Buddha did in His previous life for other beings, out of compassion.