How is "binding and loosing" extrapolated to all clergy?

Christ gave the power of "binding and loosing" to Peter alone with this saying; however, a little while later He also told all the apostles:

Amen, I say to you, whatever you bind on earth shall be bound in heaven, and whatever you loose on earth shall be loosed in heaven.

(Matthew 18:18)

Similarly, Paul states that

All this is from God, who has reconciled us to himself through Christ and given us the ministry of reconciliation.

(2 Corinthians 5:18; emphasis added)

For these reasons, the Church believes,

our Lord placed Simon alone as the rock and the bearer of the keys of the Church, and made him shepherd of the whole flock; it is evident, however, that the power of binding and loosing, which was given to Peter, was granted also to the college of apostles, joined with their head.

(Lumen Gentium, Section 22)

The Catechism adds,

Christ has willed that in her prayer and life and action his whole Church should be the sign and instrument of the forgiveness and reconciliation that he acquired for us at the price of his blood. But he entrusted the exercise of the power of absolution to the apostolic ministry which he charged with the "ministry of reconciliation." The apostle is sent out "on behalf of Christ" with "God making his appeal" through him and pleading: "Be reconciled to God."

So what allows us to believe that this power was passed on from the apostles to their successors? The same thing that allows us to believe that their successors were intended to take their place and exercise exactly their powers with those whose spiritual welfare they were intended to oversee:

In order that the mission entrusted to them might be continued after their death, [the apostles] consigned, by will and testament, as it were, to their immediate collaborators the duty of completing and consolidating the work they had begun, urging them to tend to the whole flock, in which the Holy Spirit had appointed them to shepherd the Church of God. They accordingly designated such men and then made the ruling that likewise on their death other proven men should take over their ministry. ... Just as the office which the Lord confided to Peter alone, as first of the apostles, destined to be transmitted to his successors, is a permanent one, so also endures the office, which the apostles received, of shepherding the Church, a charge destined to be exercised without interruption by the sacred order of bishops.

(Lumen Gentium section 20, quoted in Catechism of the Catholic Church, paragraphs 861–62)

So where do the priests get that power? Priests are appointed to assist bishops, and share in their power as it is delegated to them:

The function of the bishops’ ministry was handed over in a subordinate degree to priests so that they might be appointed in the order of the priesthood and be co-workers of the episcopal order for the proper fulfillment of the apostolic mission that had been entrusted to it by Christ.

(Presbyterorum Ordinis, paragraph 2 section 2; quoted in Catechism of the Catholic Church paragraph 1562)

This then is the answer: Jesus Christ gave this power to all his apostles, and then in order to safeguard the Church, gave them (through the Holy Spirit) the ability to appoint successors who carried on all their powers and responsibilities. These men in turn appointed delegates (priests) who were able to exercise some of those powers where the bishop might not be available to.

Christ gave St. Peter the "power of the keys" to bind or loose.

Addressing the question of "Whether priests alone have the keys?," St. Thomas Aquinas writes:

There are two kinds of key. One reaches to heaven itself directly, by remitting sin and thus removing the obstacles to the entrance into heaven; and this is called the key of "order." Priests alone have this key, because they alone are ordained for the people in the things which appertain to God directly. The other key reaches to heaven, not directly but through the medium of the Church Militant. By this key a man goes to heaven, since, by its means, a man is shut out from or admitted to the fellowship of the Church Militant, by excommunication or absolution. This is called the key of "jurisdiction" in the external court, wherefore even those who are not priests can have this key, e.g. archdeacons, bishops elect, and others who can excommunicate. But it is not properly called a key of heaven, but a disposition thereto.

Addressing the question of "Whether holy men who are not priests have the keys?," St. Thomas explains why only those men ordained to the priesthood have the power of the keys:

There is this difference between a principal and an instrumental agent, that the latter does not produce, in the effect, its own likeness, but the likeness of the principal agent, whereas the principal agent produces its own likeness. Consequently a thing becomes a principal agent through having a form, which it can reproduce in another, whereas an instrumental agent is not constituted thus, but through being applied by the principal agent in order to produce a certain effect. Since therefore in the act of the keys the principal agent by authority is Christ as God, and by merit is Christ as man, it follows that on account of the very fulness of Divine goodness in Him, and of the perfection of His grace, He is competent to exercise the act of the keys. But another man is not competent to exercise this act as principal agent, since neither can he give another man grace whereby sins are remitted, nor can he merit sufficiently, so that he is nothing more than an instrumental agent. Consequently the recipient of the effect of the keys, is likened, not to the one who uses the keys, but to Christ. Therefore, no matter how much grace a man may have, he cannot produce the effect of the keys, unless he be appointed to that purpose by receiving [holy] orders [(the priesthood)].

Thus, your question really amounts to why St. Peter and his successors, the popes, are necessary for there to be a valid priesthood.