Who was the first person ordered executed by a Roman Catholic pope?

The first person formally condemned to death by the papacy was Arnold of Brescia (c. 1090 – June 1155), an Italian canon regular from Lombardy. He called on the Church to renounce property ownership and participated in the failed Commune of Rome.

Exiled at least three times and eventually arrested, Arnold was hanged by the papacy, then was burned posthumously and (his ashes) thrown into the River Tiber. Though he failed as a religious reformer and a political leader, his teachings on apostolic poverty gained currency after his death among "Arnoldists" and more widely among Waldensians and the Spiritual Franciscans, though no written word of his has survived the official condemnation. Protestants rank him among the precursors of the Reformation.

After Pope Eugene's death, Pope Adrian IV swiftly took steps to regain control of Rome. He allied with Frederick Barbarossa, who took Rome by force in 1155 after a Holy Week interdict and forced Arnold again into exile. Arnold was seized by Imperial forces and tried by the Roman Curia as a rebel. Importantly, he was never accused of heresy. Faced with the stake, he refused to recant any of his positions. Convicted of rebellion, Arnold was hanged in June and his body burnt. Because he remained a hero to large sections of the Roman people and the minor clergy, his ashes were cast into the Tiber, to prevent his burial place becoming venerated as the shrine of a martyr.

In 1882, after the collapse of Papal temporal powers, the city of Brescia erected a monument to its native son. - Arnold of Brescia Wikipedia)

The first known persons directly ordered to be executed by the a pope himself seems to be the brothers Cardinal Carlo Carafa and Giovanni Carafa, Duke of Paliano, nephews of Paul IV, sentenced to strangulation in prison and beheading, respectively, by Pius IV, as his first public act (March 5, 1561).

Giovanni Carafa (died 5 March 1561), Duke of Paliano, was a papal nephew and minor Italian prince.

After the failure of the Pope's war with Spain in 1558, their notoriety became a liability, and they were banished from Rome on 27 January 1559. Paul IV died in August of that year, and Giovanni and Carlo were put on trial by the new Pope, Pius IV, in July 1560. The trial's proceedings were concluded in March 1561 when, under sealed orders of the Pope, the brothers were executed in Rome. Carlo, as a cardinal, was strangled in the Castel San Angelo, while, two days later, Giovanni was beheaded at the Tor di Nona prison along with two companions. The sentence was overturned under the next pope, Pius V, in 1567, after a petition by their surviving brother, and their prosecutor was executed for having deceived Pius IV. Giovanni Carafa, Duke of Paliano

As for the Cardinal Callo Carafa, Wikipedia reads as follows:

Carlo Carafa (29 March 1517 - 6 March 1561) of a distinguished family of Naples, vicious and talented3 was successively condottiero in the service of France and of Spain, vying for their protectorates in Italy until 1555, when he was made a cardinal,3 to 1559 the all-powerful favourite and Cardinal Nephew of Pope Paul IV Carafa, whose policies he directed and whom he served as papal legate in Paris, Venice and Brussels. According to the Jesuit, later Cardinal, Francesco Sforza Pallavicino, writing the history of the Council of Trent, his subtlety of spirit and grace of address, physical courage and instinct for glory were overridden by his insatiable thirst for power.

In June 1560, Paul's successor, Pope Pius IV arrested the leading members of the family - Carlo, his brother Duke Giovanni, and their nephew the Cardinal Archbishop of Naples, seizing their papers, and levying a range of charges relating to abuses of power during Paul's reign. Carlo was charged with murder, sodomy, and the promotion of Protestantism. After a nine-month trial, he was condemned along with his brother, and was executed by strangulation at Castel Sant' Angelo on the night of 6 March 1561. His execution was considered at the time to have been motivated primarily by political factors such as his pro-French, anti-Spanish policies.

On September 26, 1567, the sentence was declared unjust by Pope Pius V. The memory of the victims was vindicated and their estates restored. Carlo Carafa