What is the distinction between a sacrament and an ordinance?

It is not only Catholics who believe in sacraments. All Orthodox and most Protestants believe in them as well, but with some differences between them. It is only Baptists and their offspring which eschew the word sacrament and prefer ordinance.

The word sacrament comes from the Latin sacramentum, which is itself a translation of the Greek mysterium. These words mean "mystery," which is a concept in the Bible too complex and controversial to get into adequately here, but the way it relates to the word sacrament is that the sacraments of the church mysteriously and invisibly convey grace to the recipients. Catholics, Lutherans, and Presbyterians all define "sacrament" similarly:

The sacraments are efficacious signs of grace, instituted by Christ and entrusted to the church, by which divine life is dispensed to us. The visible rites by which the sacraments are celebrated signify and make present the graces proper to each sacrament. They bear fruit in those who receive them with the required dispositions. (Catechism of the Catholic Church, 1131).

Sacraments [are] rites which have the command of God, and to which the promise of grace has been added. (Defense of the Augsburg Confession, Article XIII. (VII)).

Sacraments are holy signs and seals of the covenant of grace, immediately instituted by God, to represent Christ and his benefits, and to confirm our interest in him: as also to put a visible difference between those that belong unto the Church, and the rest of the world; and solemnly to engage them to the service of God in Christ, according to his Word. There is in every sacrament a spiritual relation, or sacramental union, between the sign and the thing signified; whence it comes to pass that the names and effects of the one are attributed to the other. The grace which is exhibited in or by the sacraments, rightly used, is not conferred by any power in them; neither doth the efficacy of a sacrament depend upon the piety or intention of him that doth administer it, but upon the work of the Spirit, and the word of institution, which contains, together with a precept authorizing the use thereof, a promise of benefit to worthy receivers. (Westminster Confession of Faith, Chapter 27).

Catholicism considers the following to be sacraments: baptism, Eucharist, confirmation, penance, anointing of the sick, marriage, and Holy Orders (ordaining priests and such). Presbyterians only consider baptism and the Lord's Supper (what Catholics call the Eucharist) sacraments, and Lutherans consider, baptism, the Lord's Supper, and penance to be sacraments. Protestants believe in marriage and confession of sins and may or may not believe in the other things Catholics call sacraments -- they just don't call them sacraments.

According to one Eastern Orthodox source, "Usually seven sacraments are counted: [the same seven as in Catholicism.] The practice of counting the sacraments was adopted in the Orthodox Church from the Roman Catholics. It is not an ancient practice of the Church and, in many ways, it tends to be misleading since it appears that there are just seven specific rites which are 'sacraments' and that all other aspects of the life of the Church are essentially different from these particular actions. The more ancient and traditional practice of the Orthodox Church is to consider everything which is in and of the Church as sacramental or mystical."

Baptists call baptism and the Lord's Supper "ordinances" because they are rites "ordained" by God in the Bible. This is a difference in terminology but it signals an underlying difference in theology. If you call it a "sacrament" you are confessing it to be an act done by God through his church which confers grace on the participants. If you call it an "ordinance" you are confessing it to be an act done by you the participant as a statement of your faith (in the case of baptism) or as a memorial of Christ's death (in the case of the Lord's Supper). Note that there is some overlap in these definitions, and that Presbyterians for example wouldn't object to calling the sacraments "ordinances" as well as "sacraments." But those who object to the word "sacrament" do believe that no grace is conferred objectively by the act itself.