Why did France renounce its claim on St Paul and Amsterdam Islands?

As @T.E.D. correctly points out in their comment, answering this question requires looking into French sources. In particular, the French Wikipedia article Îles Saint-Paul et Amsterdam is more detailed than the English language version. Regarding the period in question it says:

En 1842, elles éveillent l’intérêt du Polonais Adam Mierosławski (pl), capitaine du Cygne de Granville. Le capitaine Mierosławski a passé son diplôme de capitaine au long cours sous le nom de son frère Pierre Louis Adam Mierosławski, en utilisant le passeport français de ce dernier. En 1843, Adam Mierosławski propose au gouverneur de l’île Bourbon (Réunion), le contre-amiral Bazoche, la prise de possession de ces îles désertes. En absence de navire de guerre en rade, Bazoche fait appel au trois-mâts L'Olympe, commandé par Martin Dupeyrat. Le capitaine Dupeyrat et son bateau vont ramener Adam Mierosławski sur ces îles. Le capitaine Mierosławski est mandaté par le gouverneur de Bourbon, par l’arrêté du 8 juin 1843, pour assumer le commandement de ces îles aussitôt la prise de possession au nom de la France.

In 1842, they aroused the interest of the Polish Adam Mierosławski (pl), captain of the Cygne de Granville. Captain Mierosławski graduated as a long-distance captain under the name of his brother Pierre Louis Adam Mierosławski, using the latter's French passport. In 1843, Adam Mierosławski proposed to the governor of Bourbon Island (Reunion), Rear Admiral Bazoche, to take possession of these deserted islands. In the absence of a warship in the harbour, Bazoche called on the three-master L'Olympe, commanded by Martin Dupeyrat. Captain Dupeyrat and his boat will bring Adam Mierosławski back to these islands. Captain Mierosławski is mandated by the governor of Bourbon, by the decree of June 8, 1843, to assume command of these islands as soon as possession is taken in the name of France.

Entre-temps le Royaume-Uni conteste cette prise de possession. Pour éviter les problèmes et au vu de la pauvreté des deux îles, la France envoie une dépêche à l’amiral Bazoche demandant le rappel de la garnison. Mierosławski le conteste (il menace même de hisser le drapeau polonais!). Il commence alors à négocier avec Bazoche, un ami, Adolphe Camin et d'autres interlocuteurs de la Réunion avec qui ils fondent une société par actions en 1845 pour l'exploitation des deux îles et la création d'établiss*m*nts sur place. L'île Saint-Paul, où est alors installée une pêcherie, compte pendant la période qui suit jusqu'à 40 habitants, mais privée de son fondateur et désertée peu à peu par les pêcheurs, l'entreprise s'interrompt en 1853.

In the meantime, the United Kingdom is contesting this takeover. To avoid problems and in view of the poverty of the two islands, France sends a dispatch to Admiral Bazoche requesting the recall of the garrison. Mierosławski disputes this (he even threatens to raise the Polish flag!). He then began to negotiate with Bazoche, a friend, Adolphe Camin and other interlocutors from Reunion with whom they founded a joint-stock company in 1845 for the exploitation of the two islands and the creation of establishments on the spot. Saint-Paul Island, where a fishery was then installed, had up to 40 inhabitants during the following period, but deprived of its founder and gradually deserted by fishermen, the business was interrupted in 1853.

I leave to Google translate to provide a detailed translation, but in a nutshell - the islands didn't represent any particular military or commercial interest and taking their possession was an initiative of a single man - Adam Mierosławski, a Polish commander of a French ship (he also had a French passport.) While in 1843 the French government does respond positively to this initiative, it does not want to make much effort in defending the islands, when their possession is contested by the United Kingdom, the very same year. Mierosławski is not satisfied with this situation and organizes a society aiming at the development of the two islands, which results in a fishery installed on Saint-Paul. The population then grows up to 40 inhabitants, but the fishery eventually ceases to exist in 1853, and the island is de facto abandoned this year. It is only in 1871 that the French return to the islands and their development and serious exploration begin.