Is the use of rosary by Catholics a pagan practice?

No, the rosary is not of pagan origin.

St. Louis-Marie Grignion de Montfort concisely narrates the institution of the rosary in his The Secret of the Rosary (pt. 1 "What the Rosary Is", "1^st Decade: Origin and Name", "Second Rose: Origin [of the Rosary]"):

SINCE the Holy Rosary is composed, principally and in substance, of the Prayer of Christ and the Angelic Salutation, that is, the Our Father and the Hail Mary, it was without doubt the first prayer and the first devotion of the faithful and has been in use all through the centuries, from the time of the Apostles and disciples down to the present.

But it was only in the year 1214, however, that Holy Mother Church received the Rosary in its present form and according to the method we use today. It was given to the Church by Saint Dominic who had received it from the Blessed Virgin as a powerful means of converting the Albigensians and other sinners.

I will tell you the story of how he received it, which is found in the very well-known book De Dignitate Psalterii by Blessed Alan de la Roche. Saint Dominic, seeing that the gravity of people’s sins was hindering the conversion of the Albigensians, withdrew into a forest near Toulouse where he prayed unceasingly for three days and three nights. During this time he did nothing but weep and do harsh penances in order to appease the anger of Almighty God. He used his discipline so much that his body was lacerated, and finally he fell into a coma.

At this point Our Lady appeared to him, accompanied by three Angels, and she said:

“Dear Dominic, do you know which weapon the Blessed Trinity wants to use to reform the world?”

“Oh, my Lady,” answered Saint Dominic, “you know far better than I do because next to your Son Jesus Christ you have always been the chief instrument of our salvation.”

Then Our Lady replied:

“I want you to know that, in this kind of warfare, the battering ram has always been the Angelic Psalter which is the foundation stone of the New Testament. Therefore if you want to reach these hardened souls and win them over to God, preach my Psalter.”

So he arose, comforted, and burning with zeal for the conversion of the people in that district he made straight for the Cathedral. At once, unseen Angels rang the bells to gather the people together and Saint Dominic began to preach.

At the very beginning of his sermon an appalling storm broke out, the earth shook, the sun was darkened, and there was so much thunder and lightning that all were very much afraid. Even greater was their fear when looking at a picture of Our Lady exposed in a prominent place they saw her raise her arms to Heaven three times to call down God’s vengeance upon them if they failed to be converted, to amend their lives, and seek the protection of the Holy Mother of God.

God wished, by means of these supernatural phenomena, to spread the new devotion of the Holy Rosary and to make it more widely known.

At last, at the prayer of Saint Dominic, the storm came to an end, and he went on preaching. So fervently and compellingly did he explain the importance and value of the Holy Rosary that almost all the people of Toulouse embraced it and renounced their false beliefs. In a very short time a great improvement was seen in the town; people began leading Christian lives and gave up their former bad habits.

For more details on the history of the rosary, see Willam's The Rosary: Its History and Meaning

Is the use of rosary by Catholics a pagan practice?

The short answer is no.

There are several points to be taken into consideration here.

First of all, let us look at the point that many claim the rosary to be vain repetition. The origins of the 150 beads of the original rosary are the number of Psalms in the Bible. It is the poor man's way of praying the Psalms (substituting a prayer for psalm when one could not read).

Prayers with beads like the rosary may have begun as a practice by the laity to imitate the Christian monasticism of the Liturgy of the Hours, during the course of which the monks prayed the 150 Psalms daily. As many of the laity and even lay monastics could not read, they substituted 150 repetitions of the Our Father (Pater Noster in Latin) for the Psalms, sometimes using a cord with knots on it to keep an accurate count. - History of the Rosary (Wikipedia)

But praying such is not vain repetition as one reads in Matthew 6:7.

Matthew 6:7 KJV But when ye pray, use not vain repetitions, as the heathen do: for they think that they shall be heard for their much speaking.

Other translations do not even mention "vain repetitions" as found in this question on Biblical Hermeneutics: What, exactly, counts as a “vain repetition” (Matthew 6:7)?

There are differing views on the history of the Rosary. The exact origin of the Rosary as a prayer is less than clear and subject to debate among scholars. The use of knotted prayer ropes in Christianity goes back to the Desert Fathers in the 3rd and early 4th centuries. These counting devices were used for prayers such as the Jesus prayer in Christian monasticism. The period after the First Council of Ephesus in 431 witnessed a gradual growth in the use of Marian prayers during the Middle Ages.

Let us not forget that the Orthodox and Eastern Catholic Rites use a Prayer Rope as an aide in their praying.

The history of the prayer rope goes back to the origins of Christian monasticism itself. When monks began going into the deserts of Egypt, it was their custom to pray the entire 150 Psalms every day. However, because some of the monks were unable to read, they would either have to memorize the psalms or perform other prayers and prostrations in their stead. Thus the tradition of saying 150 (or more) Jesus Prayers every day began.

The western Rosary is descendent and developed from this same tradition.

The invention of the prayer rope is attributed to Saint Pachomius in the fourth century as an aid for illiterate monks to accomplish a consistent number of prayers and prostrations in their cells. Previously, monks would count their prayers by casting pebbles into a bowl, but this was cumbersome, and could not be easily carried about when outside the cell. The use of the rope made it possible to pray the Jesus Prayer unceasingly, whether inside the cell or out, in accordance with Saint Paul's injunction to "Pray without ceasing" (I Thessalonians 15:17).

It is said that the method of tying the prayer rope had its origins from the Father of Orthodox Monasticism, Saint Anthony the Great. He started by tying a leather rope with a simple knot for every time he prayed Kyrie Eleison ("Lord have Mercy"), but the Devil would come and untie the knots to throw off his count. He then devised a way—inspired by a vision he had of the Theotokos—of tying the knots so that the knots themselves would constantly make the sign of the cross. This is why prayer ropes today are still tied using knots that each contain seven little crosses being tied over and over. The Devil could not untie it because the Devil is vanquished by the Sign of the Cross.