What were the causes of decline of Babylonian numeric system?

Simply put, the decimal system is more convenient for most types of calculations. As you point out, there are systems that still use base 60. And there are others such as binary and hexadecimal which are applied in other areas where they are applicable.

But the main reason for its decline is the unwieldiness. 60 as a base is difficult to use because you have to remember at least 59 unique names ( as opposed to decimal where twenty-one, thirty-one, forty-one etc follow the same pattern). To mitigate these difficulties, arithmeticians looked for a smaller number. This led to the decline of the 60 base. See this page 97.

It is also important to note that various geographies and cultures have used various systems over time, and the sexagesimal system is predated by various other systems:

"Historically, finger counting, or the practice of counting by fives and tens, seems to have come later than counter-casting by twos and threes, yet the quinary and decimal systems almost invariably displaced the binary and ternary schemes. A study of several hundred tribes among the American Indians, for example, showed that almost one-third used a decimal base, and about another third had adopted a quinary or a quinary-decimal system; fewer than a third had a binary scheme, and those using a ternary system constituted less than 1 percent of the group. The vigesimal system, with the number 20 as a base, occurred in about 10 percent of the tribes."

"An interesting example of a vigesimal system is that used by the Maya of Yucatan and Central America. This was deciphered some time before the rest of the Maya languages could be translated. In their representation of time intervals between dates in their calendar, the Maya used a place value numeration, generally with 20 as the primary base and with 5 as an auxiliary."

See this

So it seems a balance between a "small" number - already known- such as 5, and a slightly larger 12, as base was ultimately arrived at- the number being 10.

Another point to add to your question:

"The Mesopotamian civilizations of antiquity are often referred to as Babylonian, although such a designation is not strictly correct. The city of Babylon was not at first, nor was it always at later periods, the center of the culture associated with the two rivers, but convention has sanctioned the informal use of the name “Babylonian” for the region during the interval from about 2000 to roughly 600 BCE.When in 538 BCE Babylon fell to Cyrus of Persia, the city was spared, but the Babylonian Empire had come to an end. “Babylonian” mathematics, however, continued through the Seleucid period in Syria almost to the dawn of Christianity."

Edit: Additional Information on "Babylonian" Mathematics

Much of Babylonian Math tablets comes from "House F", a scribal school:

House F was excavated in the first months of 1952 by a team of archaeologists from the universities of Chicago and Pennsylvania. It was their third field season in the ancient southern Iraqi city of Nippur and one of their express aims was to find large numbers of cuneiform tablets (McCown and Haines 1967, viii). For this reason they had chosen two sites on the mound known as Tablet Hill, because of the large number of tablets that had been found there in the late nineteenth century.

Several types of tablet were used for elementary schooling in Nippur, as classified by a scheme devised by Miguel Civil (e.g., 1995, 2308) to describe lexical lists—standardized lists of signs and words. But, as Niek Veldhuis (1997, 28–39) showed, this tablet typology applies equally to all elementary school exercises, including mathematical ones. It happens that mathematics has survived on just three types of tablet of from House F: the small Type IIIs and the larger Type I and IIs, of which it will be important to distinguish between the flat obverse (Type II/1) and the slightly convex reverse (II/2)...

On the multiplication tables:

The standard list of multiplications was described long ago by Neugebauer (1935–7, I 32–67; Neugebauer and Sachs 1945, 19–33) and is very well known.

...

The series starts with a list of one- and two-place reciprocal pairs, encompassing all the regular integers from 2 to 81. It is followed by multiplication ‘tables’ for sexagesimally regular head numbers from 50 down to 1 15, with multiplicands 1–20, 30, 40, and 50.

....

Returning to the standard series of multiplications as attested in House F, nine of the 40 known head numbers—namely 48, 44 26 40, 20, 7 12, 7, 5, 3 20, 2 24, and 2 15—do not survive on known tablets. Should we attribute these omissions to the accidents of recovery or to deliberate exclusion from the series? The patterns of attestation make it easier to make de[ nitive statements about the higher head numbers than the lower. The head number 48, for instance, is included in just five of the 71 ‘combined’ tables catalogued by Neugebauer (two of those five are from Nippur), compared to 23 certain omissions. He lists no ‘single’ tables for 48. Similarly, 2 15 occurs in two out of nine possible ‘combined’ tables, neither of them from Nippur, and in no ‘singles’. It is not surprising, therefore, that the 48 and 2 15 times tables were apparently not taught in House F. The exclusion of 44 26 40, is rather more surprising: given its place near the start of the standard series it is presumably not simply missing by archaeological accident. On the other hand none of Neugebauer’s ‘combined’ tables appear to omit it, while he lists three ‘single’ tables for 44 26 40. 9 is is a deliberate but idiosyncratic omission then, particular to House F—though perhaps a judicious one; none of the other head numbers are three sexagesimal places long.

Unsuccessful methods and mistakes by Students:

The other two calculations identified so far on House F tablets are also attempts to find reciprocals, but conspicuously less successful than the first.

...

As in our first example, the student has split 4 37;46 40 into 4 37; 40 and 0;06 40. He has appropriately taken the reciprocal of the latter—9—and multiplied it by the former, adding 1 to the result. However, instead of arriving at 41 39 + 1 = 41 40, our student has lost a sexagesimal place and found 41;39 + 1 = 42;39. Unable to go further with his calculation (for the next stage is to find the reciprocal of the number just found, but his is coprime to 60) he has abandoned the exercise there. The correct answer would have been 0;00 12 57 36. the last calculation of the three is the most pitiful ... the student has got no further than... On the other hand, 4 37 46 40 does not, as far as I can ascertain, fit the pattern; presumably it was chosen because, like the other two, it terminates in the string 6 40. One possible interpretation of this commonality is that three students were set similar problems at the same time, using a common method and a common starting point but requiring different numerical solutions.

It is clear from these that the techniques were not perfected yet- even if we today believe that a base 60 math was a viable system.