Evolution of destroyers

Since you're asking on History SE, remember the most important rule:

Put your sources into context: Who wrote them and why?

It might seem odd to apply this rule to something as objective as the top speed of warships, but it remains true even here.

• In the years before World War One, quite a lot of money for destroyer builders rode on making or exceeding their design speed. To make this as objective as possible, speed was measured over one of several "measured miles" in sheltered water, with hand-picked engineers and stokers and hand-picked coal. Often the engines were worn out by the time a destroyer made its trial speed. It wasn't fraud -- everybody understood that trial speed wasn't the same as operational speed on the North Sea.
• Between the world wars, naval arms limitation treaties made many powers lie about their ships. The London naval treaty defined destroyers as ships up to 1,850 tons with guns up to 5.1 inch. Anything bigger would have counted as a cruiser. While lies about the gun caliber were difficult, the displacement left some room for trickery.
• For modern warships, what makes you think that navies tell the internet and any possible enemies the truth? For decades, the US Navy would give the performance of their submarines as "over 20 knots, over 200 feet depth." True, except that it was well over 20 knots and well below 200 feet. In recent years they decided to admit the speed is "over 25 knots." Independent guesstimates put them over 35 knots. It may be that the published data on surface ships is closer to the truth. Or not. Note that the top speed of American carriers is given as "over 30 knots" as well.

Follow-Up: I'm saying that early speeds were systematically overstated and that current top speeds show interesting patterns which might suggest disinformation or classification.

• Early Destroyers: David K. Brown, The Grand Fleet, Appendix 2, and Warrior to Dreadnought, Chapter 8.
• Treaties: The Tomozuru, noting that it was a torpedo boat and not a destroyer.
• Modern Destroyers, Clancy questions the numbers on subs in his non-fiction book Submarine. Compare the listed speeds for CVN-79, CGN-38, DDG-1000.

There has been a huge shift in naval battle concepts during the 20th century. This is especially true of the 1960's, and the advent of missile cruisers.

When ships fought with guns, there was a matter of "catching" a ship, which could escape if it was faster.

Since the 1960's, ships fight with guided missiles (French Exocets then, American Harpoons now). These are incredibly fast, and can cruise well over 100 Nautical Miles, and even be guided over the horizon by air support. Since this is the case, the added 10Kts of speed are simply not a factor that will justify the fuel expenditure.

If you look at patrol boats, which spend a lot of time chasing various things as part of their role, you'll find that many of them go well above 40 Kts, even as fast as 47 Kts.

Why the top speed of the fastest (surface, displacing) ships shows such a strange pattern of evolution?

"Speed is armor" is the idea that you can catch anything you can sink, and you can run away from anything that can sink you. There's a lot of things that can sink a destroyer, so they were fast. While ranges were short, weapons were inaccurate, and detection was done with eyeballs this made sense. Missiles and radar changed all that.

In WWI fire control was poor and ranges were short, but ship speeds suddenly increased, and the torpedo was a dangerous new equalizer, albeit very short ranged.

The introduction of the modern Dreadnought style battleship and battlecruiser saw a rapid increase in the speed of capital ships from 10 to 12 knots to 20 to 25 knots. Destroyers had to keep up with the fleet to protect them from torpedo boats, and they needed the speed to catch the torpedo boats.

They also needed speed and agility to quickly get close to a capital ship, typically a few thousand yards, launch their torpedoes, and run away. The very poor fire control and accuracy in those days meant a destroyer could get away with that. The idea that "speed is armor" worked for small ships.

Early WWII saw faster, more accurate, and longer ranged guns and torpedoes, but still manually aimed. "Speed is armor" still won the day for a small, maneuverable ship. Battle fleets were even faster with carriers going over 30 knots. To fulfill their traditional role, destroyers' speed also had to increase.

Late WWII saw the introduction of radar range finding and fire control. Gunfire was becoming more accurate and it became more difficult for destroyers to dash in and launch torpedoes at capital ships. One of the last hurrahs for the classic destroyer attack is the Battle off Samar where a handful of destroyers, escort destroyers, and escort carriers convinced a heavy Japanese capital fleet to retreat.

Post WWII saw the introduction of ubiquitous radar and the guided missile. This changed both the offensive and defensive role of destroyers.

Search radar allows ships to detect destroyers at long distances, even in bad weather. Airborne search radar can even do it over the horizon. No longer can a destroyer sneak up to a capital ship and torpedo it.

Fortunately they don't have to. The slow, short ranged torpedo was no longer the equalizer. Now it was the missile. A destroyer can now launch Harpoon missiles from 50 miles out and potentially cripple a capital ship. There's no longer a need to dash in close and dash out.

And when getting shot at by a computer guided gun or missile, a difference of 5 or 10 knots isn't going to matter.

Now the space and weight is used for other things. Better efficiency, better accommodations, more computers, more electronics, and more power for all those computers and electronics.

To be perfectly correct, the Cossack and the ships from the Adamss on are not true "Destroyers" as the true destroyers first came about in the 1890s as an invention of Jackie Fisher named "torpedo-boat destroyers". On this line, the more modern "destroyers" are not what can be classified as classic destroyers but as cruisers (The last of the classic DDs were the Gearing Class).

From what I see in the ships post-Gearing, the DD and the Cruiser design merged and the cruiser-type design took precedence. The modern terms of "Destroyer" and "Cruiser" have more to do with the roles that the ships play rather than anything from their historical antecedents. I would like to point out that the Spruance DDs and the Ticonderoga CGs are just different superstructures on the same hull.

TL;DR

The speed on the later ships are more in line with the cruiser doctrine as the designs of those ships are basically cruisers.

The pre-Adams class ships on your list are TBDs and the ones from the Adams onward are modified cruisers. Cruisers have a different usage doctrine than Torpedo-Boat Destroyers have.

As time has gone by the distinction between the name for a naval role and the class of ships that performed that role has shifted.

In the age of sail, there were essentially three classes of warship; Ships of the line (of battle), which were large warships capable of holding their own in a fleet action (typically with 50 or more guns), Frigates, which were smaller but still potent warships (with 20 to 50 guns) and the final class which was anything smaller than a frigate - from twin masted brigs, right down to gunboats (which were a rowing boat with a cannon in the bows). As the age of sail progressed, and as improved technology allowed, the vessels of all three classes grew in size (tonnage) and firepower.

At that time the cruiser was not a class of ship but a role. A "cruiser" was any ship that operated independently of a fleet. Typically, this role was filled by a frigate of some form (due to their sailing qualities) but it could also be applied to a smaller or larger vessel (it wasn't that rare for smaller ships of the line to be sent on a cruise).

By the time of the transition from sail to steam power (and the almost simultaneous transition from wood to metal construction), the distinction between a large frigate and a small ship of the line had become very blurred. The first fully armored warships were notionally frigates but their improved protection and firepower made them a match for almost any vessel then at sea. It was about this time that the term cruiser shifted from being just a role to being a class of vessel built to perform that role.

The introduction of the modern self-propelled torpedo gave small steam-powered vessels the ability to punch way above their weight. Small, fast, agile torpedo boats could use their speed to get in close to a fleet and their torpedos could, theoretically, sink a ship of any size. As a response, the torpedo-boat destroyers were developed. These were specialized ships that were intended to use their own speed to intercept the torpedo boats before they could get into range of the fleet. The need for improved seaworthiness to stay with the fleet while keeping their speed lead to these destroyers increasing in size.

With the introduction of torpedos to both the destroyers themselves and to submarines, the threat from enemy torpedo boats shifted to enemy submarines and destroyers. So the destroyer's role shifted from being torpedo boat hunters to being submarine and destroyer hunters. As is usual with warship development, the trend is to increase the firepower of any class and this increase is usually also accompanied by an increase in size. By the end of WW1, the destroyer was the most common class of warship and this lead to them becoming much more multipurpose (similar to the frigates of the age of sail). Their role as submarine hunters was largely taken over by smaller, cheaper, more specialized vessels (termed as "corvettes" or "destroyer escorts") which were about the same size as the original torpedo boat destroyers. As they changed role, the need for the raw speed which they needed as interceptors diminished.

After WW2, the introduction of the guided missile gave even small warships the same range and firepower as the largest vessels. The ability to carry one or more helicopters also gave them expanded capabilities (especially in the anti-submarine role). Therefore the distinction between a frigate, a destroyer and a cruiser was less one of purpose and simply one of physical size. Even this distinction is blurred when comparing ships between different navies. A ship that is labelled as a destroyer in one fleet might be termed a frigate in another country's fleet.