score:14
Why yes, indeed.
I count, fast and dirty, at least 37 Japanese destroyers sunk by US submarines in WW2. A good place to look is the Joint Army Navy Assessment Committee though it requires a little perseverance. At least 9 US destroyers or destroyer escorts were sunk by German submarines and at least 8 sunk by Japanese submarines (including midgets and Kaitens). These can be found in the easiest format here. Wiki - US Navy Losses.
Pretty sure the Royal Navy and the Royal Netherlands Navy submarines and other Allied navies had their share of enemy destroyers as well. And the same lost their share of destroyers to Germans, Italian or Japanese submarines, for example perhaps here: Wiki - Royal Navy Losses.