Why were troops with bayonets often effective against enemy cavalry even though the bayonet was a "secondary" weapon?

I think there were two reasons at play:

1. Musketeers were "formation troops", i.e., they were used to fighting in a close formation, standing next to each other and acting in sync. This means that when they affixed bayonets and pointed them at the cavalry, the horses were facing tightly knit line of steel and could not carry their riders close enough for their sabers to be effective. The spear/halberd carriers were more accustomed to one-on-one combat (during middle ages), so they did not always managed to stand shoulder to shoulder to meet the enemy cavalry in time for its attack.
2. Cavalry which attacked spear/halberd carriers was the heavy knight cavalry: armored man and horse, with the rider armed with a lance or long sword. The cavalry which faced musketeers was lighter (at most a cuirass) and their weapons were shorter.

Both changes were happening during the same time period, driven by the gunpowder revolution which made body armor useless and forced close formation on the infantry (to compensate for the lack of accuracy).

What I've been missing from the current answers is that early heavy cavalry, i.e. knights in armor (mail until ~1400, plate armor only in late medieval), weren't as effective as "advertised". They surely won battles but they did as one part of a bigger army. By the point in a battle the cavalry charge was ordered, enemy formations have already been "loosened" up by archer's volleys. However archers weren't supposed or even allowed to win battles. When they had done their job, they were ordered to the back so that cavalry could sack the win. Keep in mind that heavy cavalry was composed of nobles, while infantry or archers were commoners. The glory of the win would not be handed to commoners.

As a reference, look at the accomplishment of England, which was to put up more (in quality and quantity) trained archers. Suddenly knights were losing battles against archers. The rules of the game had changed and obviously they wouldn't change back. In this the story of Robin Hood, a noble archer, can be seen as a kind of PR-measure for archers. It's not surprising, that Robin Hood is an English tale.

Musketeers with bayonets are just a way to combine ranged and close-combat capabilities of archers and spearmen. However the actual beginning decline of cavalry was earlier with English longbow-men.

To complete the story of decline, what defeated the cavalry in the end was barbed wire.

Short answer: The bayonet wasn't really a "secondary" weapon, even though it "followed" the musket. A musket with a bayonet was approximately the same length and weight as a spear, and could function in that capacity. Except that these "spears" could also shoot.

Bayonets gave musketeers a decisive advantage on open ground over other users of missile weapons that didn't have bayonets. This disadvantage held for e.g. American musketeers without bayonets early in the American Revolution, even though the Americans were (by far) the better shots. The Americans might well have been beaten by spearmen as well, but the "single shot" muskets acted as a morale stabilizer in giving British troops the feeling, "Our guns give us a "fair chance" in the shooting, while their chances "up close and personal" were more than "fair."

Likewise, bayoneters acted like spearmen against cavalry, except that the advantage of the "first shot" might have stiffened their morale. And the presence of the guns slowed the momentum of the first charge while preventing the cavalrymen from using armor.

I wouldn't say that knights were ineffective. True, you can waste your warriors in ineffective charge, but so you can do with your archers shooting out of range, or at heavy plated soldiers(war bow would not penetrate plate- they did it with mild steel and proper, hardened steel, and unless you got one in a visor, late XVth century armor was good to go). Then bow-lovers get all offended and ask, why bows were used at all then. Well heavy knights were only about 10-20 % of forces on the battlefield. Other people didn't have such good protection. And horses often weren't covered in plate...

XVIth-XVTIth century cavalry of the West was often lighter cavalry (not always, but quite often only armor they carried was cuirass and helmet, maybe some gauntlets) they main purpose on the battlefield was to support pike and shot formations and harass enemy(they often carried short fire arms), and then, when enemy was finally scattered, charging in. There is a number of factors at work here- why war was like that.

1. On average pike was longer than average knights lance, so horse would be killed before knight could reach opponents, not to mention, even if he got one pikemen, he would find himself in a forest of other pikes.

2. Training, say, a hundred pikemen and musketeers was still cheaper than giving as a reward land and titles to a knight(and by that i mean a knight, knighted individual, not a guy in armor on the horse- that one was called man-at-arms. Knights were very rare. In XIVth century in England was only about 500 real knights. In Battle of Grunwald 1410 there were only around 300 knights in army of 27000).

3. Cavalry was usually domain of wealthy- so nobility, wealthy citizens of cities and towns, and free, land owning peasants (in Poland they were called "Kmiecie", didn't find that name equivalent in English). In pike and shot era and later king was the one, who often outfitted his troops. Calvary is expensive. A row of guys with muskets is cheaper.

4. Most of the times wars were ended by taking some settlement. In other words sieges were far more prominent than field battles. To win field battle, you have to be sure of your strength. And often times that was not the case. Calvarymen aren't really good siege soldiers :) so why keep so many of them?

BUT- that is not to say that you couldn't make cavalry effective against pike and shot formations. And i will give winged hussars as an example. Battle of Kirchoml 1610- where around 3000 Polish- Lithuanian troops beat the flush out of Swedish army of 9000- 11000 in around 30 minutes. There are a number of factors at work here also.

1. Poland was very wealthy country in XVIth- XVIIth century, so average nobleman could outfit himself in elite unit gear and afford training from early age(to be a hussar, one had to train like a knight, since early age).

2. Average hussar lance was around 5- 6 meters long, longer than pike, there is a story that Lithuanian hussar killed 6 Russians with one hit of his lance in Płonka battle 1660. They also carried at least two pistols with them, so getting rid of numerical advantage of enemy wasn't so hard. They didn't just charge through. They would broke their lances, and do manoeuvre called "nazad"(basically turn around in place, forcing horse to stand on it`s hind legs and then running back) and ran back, for another lance, and another horse, if first one was winded. And they repeated charges until enemy was routing. They also weren't really heavy cavalry. They often had only cuirasses, cuisses, pauldrons, vambraces and helmets, so about as much as heaviest shooting cavalry of the West of XVIITh century. Their horses were light and rather smallish, so they could manoeuvre quickly.

3. As I said earlier, cavalry is a nobleman unit. It so happens, that nobility in XVIth XVIIth century in the West was around 2- 4% of society. Go and make them an army, especially that about half of them were women. In Polish- Lithuanian Commonwealth nobility reached around 10% of society...

So yes, you can make cavalry very effective, but there is so much factors here, that i really am not surprised that West would rather go with pike, than bastard grandchild of a knightly cavalry :)

So after this long entrance, why was bayonet so effective? Others nailed it, give or take. Average armor was given to a trooper rather than his horse, and most of them rather carried fire arms, even long ones, like Dragoons for example, than polearms. And since their war doctrine was to support and harass earlier, why would they change that? If i have only sabre or sword, guys with heavy "short spears" in dense formation will get my horse first. And combination of two types of soldiers into one (pikeman and musketeer) also made things easier for generals, as you didn't have to manoeuvre with two units.

But there are always exceptions, and ways of dealing with infantry like that. "Los Infernos Picadores" :D yep my ancestors were nutjobs. I don`t know if there is a good English language source, but basically they were a unit of polearm carrying (in polish "lanca". Knightly, and later hussar polearm was called "kopia"- "lance", while "lanca" is something between lance and spear. Just type "lanca kawaleryjska" in Google images if you are interested in how it looked like) polish cavalry known for their ferocity in battle. They became famous during Napoleon's campaign in Spain, hence their name

"What gave soldiers with bayonets (and muskets) their effectiveness?"

I'd say the main reason they were more effective against circa-18th Century cavalry, than typical pole-armed footmen were against mounted knights, was the lighter armor of the cavalry, and the attitudes and training of the time.

Medieval knights were a dedicated warrior class with heavy armor, and expected to be superior to other types of forces.

Mainly, it's quite hard to injure someone in full-body metal armor, good chain or plate, with a hand weapon. 18th Century cavalry tended to have a helmet and breastplate at most, so were much easier to wound than a fully-armored knight.

For a case study reference, this may be relevant.

This is from Battle of Waterloo, when Ney assaulted Wellington's centre in the French cavalary charge

""Initially Milhaud's reserve cavalry corps of cuirassiers and Lefebvre-Desnoëttes' light cavalry division of the Imperial Guard, some 4,800 sabres, were committed. When these were repulsed, Kellermann's heavy cavalry corps and Guyot's heavy cavalry of the Guard were added to the massed assault, a total of around 9,000 cavalry in 67 squadrons.

Wellington's infantry responded by forming squares (hollow box-formations four ranks deep). Squares were much smaller than usually depicted in paintings of the battle – a 500-man battalion square would have been no more than 60 feet (18 m) in length on a side. Vulnerable to artillery or infantry, squares that stood their ground were deadly to cavalry, because they could not be outflanked and because horses would not charge into a hedge of bayonets. Wellington ordered his artillery crews to take shelter within the squares as the cavalry approached, and to return to their guns and resume fire as they retreated."

""If infantry being attacked held firm in their square defensive formations, and were not panicked, cavalry on their own could do very little damage to them. The French cavalry attacks were repeatedly repelled by the steadfast infantry squares, the harrying fire of British artillery as the French cavalry recoiled down the slopes to regroup...."

An important aspect that seems to be neglected in many of the answers here is that while technical aspects cannot be completely dismissed, they are secondary to other concerns.

To be specific, the primary weapon of heavy cavalry is its momentum, while heavy infantry (among which musketeers from 18th century onward are counted) relies on its discipline in formation and the ability to stand its ground.

The reason why a spear, pike, halberd, or equivalently a musket with a bayonet, are a foil to cavalry is that if the infantry wielding them maintains discipline and remains in formation, cavalry cannot simply trample them and needs to resort to either harassing the formation or fighting in close quarters. If that happens, individual riders are at a disadvantage, and there is decidedly fewer of them than there is of the infantry.

Presenting an impenetrable wall of bodies and steel to a cavalry charge and thus robbing it of its momentum is in fact the way to deal with heavy cavalry. Besides ancient phalanxes, this has been successfully attempted by Scottish schiltrons, Swiss pike formations or Hussite wagon forts as early as the 13th century and spelt the end of heavy cavalry dominance on the battlefield. This was later cemented by Spanish (and later German) tercios and by the time of the Napoleonic wars, a cavalry charge against well-formed infantry was essentially hopeless.

Now, not to diminish the significance of bayonets, there are several important contributions that they, and related technologies, made.

• First, the combination of bayonet and a musket meant that you no longer needed to maintain separate missile troops and heavy infantry (where earlier it was not feasible for an archer or an arbalesteer to carry a pike around). This permitted the transition from earlier pike-and-shot formations to h*m*geneous musket infantry.
• Second, the specific technology of a socket bayonet mitigated vulnerability of infantry to sudden charges, either by cavalry or other infantry, because you could have your bayonet on and be ready to defend yourself in close combat while still being able to shoot (even if muzzleloading was made a bit more difficult by the blade sticking up from the end of your barrel). This, some historians claim, is what made the difference against tactics like the Highland Charge (see Battle of Culloden).
• Third, in comparison to earlier missile weapons, a musket was something you could train very nearly anyone to operate. Where getting reasonable proficiency with a sling would require a lifetime of training, with a bow, years, and with a crossbow months, pretty much anyone could be drilled to load and shoot a musket in a week or so. This in turn prompted the development (or more precisely, refinement) of standardized infantry drill for fighting with a bayonet, marching and maintaining unit cohesion, all of which are crucial when facing heavy cavalry.

So to conclude, it was not really the weapon that made the difference (although it helps to be able to shoot the horse from under a charging cavalryman - a lot easier and just as deadly alternative to trying to shoot the cavalryman himself), but how it was being used. Earlier heavy infantry tactics, such as pike squares, were reasonably effective when well executed, but the proliferation of the musket with a bayonet, as well as of thorough infantry drill, only finally made sure that in a head on confrontation, cavalry was rendered impotent.

As an addendum, remember that this did not make cavalry completely useless. Well into the Napoleonic wars, cavalry would play an important role in scouting as well as pursuit, and it was still possible to find infantry that was either not disciplined enough to withstand the primal terror of a cavalry charge, or was simply caught by surprise. It was also possible to get plain lucky and have the infantry square break at an inopportune moment, as happened at the Battle of Garcia Hernandez.

Furthermore, an infantry battalion that has formed square is effectively pinned down, and the massed formation makes it an easy target for artillery (because any good hit will necessarily take out more of the enemy) or infantry (on account of the square being a lot harder to miss, even with a musket). Following best practices for combined-arms tactics is vital.