score:5
Yes, the vikings did just that. A longship can have to about 100 crew/warriors on board. (Different type of ships had a smaller or larger crew on board.) A raid often was done with one or two ships. Laying siege was of course impossible. But having the element of surprise might give them the edge. Storming the walls was suicide. Favorable circumstances (fog, bad weather, etc.) might give viking raiders enough time to assault before the defenses were ready.
Surprise is everything. The city of Breda in The Netherlands was in 1590 taking from a well armed Spanish garrison with a ruse: 70 soldiers hid in a peat barge and took the city from within. So it worked very well, also after the middle ages.
Such a surprise raid wouldn't work with 1 or 2 ships on large coastal towns with harbors and fortifications. Not all towns were that large or had fortifications. Plenty of cloisters too that were close to the coast. Those could (and were) fair game to individual raiders.