Origin of the idea of a series of books

The story of Amadis de Gaula may have first been told in the 13th or 14th centuries.

The earliest known printed edition was printed in 1508, revised by Garcia Rodriguez de Montalvo, who wrote the fourth volume. It was so popular that Montalvo and other writers wrote books V through XI published from 1510 to 1546, so this could count as an early novel series.

https://en.wikipedia.org/wiki/Amad%C3%ADs_de_Gaula1

Amadis de Gaula was a medieval romance of chivalry, and hardly the first.

Earlier medieval romances include the Vulgate Cycle.

The Lancelot-Grail, also known as the Prose Lancelot, the Vulgate Cycle, or the Pseudo-Map Cycle, is a major source of Arthurian legend written in French. It is a series of five prose volumes that tell the story of the quest for the Holy Grail and the romance of Lancelot and Guinevere. The major parts are early 13th century, but scholarship has few definitive answers as to the authorship. An attribution to Walter Map is discounted, since he died too early to be the author.

The Vulgate Cycle adds an intriguing dimension to the King Arthur tradition, perpetuating Christian themes by expanding on tales of the Holy Grail and recounting the quests of the Grail knights. During this period, material takes on even more historical and religious overtones with tales that include and deal both in the death of Arthur and Merlin (drawing all the way back to Nennius's Historia Brittonum).

The Vulgate Cycle combines elements of the Old Testament with the birth of Merlin, whose magical origins are consistent with those told by Robert de Boron, as the son of a devil and a human mother who repents her sins and is baptized. Merlin is transformed into a prophet and given the ability of seeing future events by God.

The Vulgate Cycle was subject to a 13th-century revision in which much was left out and much added. The resulting text, referred to as the "Post-Vulgate Cycle", was an attempt to create greater unity in the material, and to de-emphasise the secular love affair between Lancelot and Guinevere. It omits almost all of the Vulgate's Lancelot Proper section, but includes characters and scenes from the Prose Tristan. This version of the cycle was one of the most important sources of Thomas Malory's Le Morte d'Arthur.

https://en.wikipedia.org/wiki/Lancelot-Grail2

So it took me just a few minutes to look up and copy data about two medieval series that could be considered to be book series. Depending on how closely one considers them to fit the definition of a modern book series, they could be the first novel series ever, or possibly centuries later than the first ever novel series.

The Bible is an example of such series of books. The books have mostly chronological order and share characters. They are intended to form a compact story and should not be treated alone.

The first part is "Torah", written by Moses. As Wikipedia says,

According to dating of the text by Orthodox rabbis, this occurred in 1312 BCE;[19] another date given for this event is 1280 BCE.[20]

Even if its authorship is disputed, it in my opinion does not affect the answer.

Then several authors added their parts, forming the Old Testament.

Then, about 1st century AD the process of writing was finished. Also, some other authors tried to include new things, however it was not commonly accepted.

It looks there will be no official sequel to the Bible, as the Bible itself says:

I warn everyone who hears the words of the prophecy of this scroll: If anyone adds anything to them, God will add to that person the plagues described in this scroll. (Revelation 22:18, NIV)

The story of Scherazade and The 1001 Tales of the Arabian Nights is the classic example of the captivating quality of a serial.

Doomed to marry the Sultan one day, and be executed the next as punishment for his first wife's adultery, Scherazade concocts the most brilliant scheme to survive. In (presumably) the afterglow of being deflowered, Scherazade begins to relate a wondrous tale to the Sultan, but falls asleep at the cliff-hanger before she can finish it.

Desperate to hear the end of the tale, the Sultan grants a stay on the execution for a single night so that the story can be completed. Scherazade manages to wind that into a second story that again she cannot finish before falling asleep, leaving the Sultan desperate to hear the ending.

This goes on for 1001 nights, Scherazade each night so entrancing the Sultan with the bewitching nature of the tales she relates, and the cliff-hangers that she is able to endlessly concoct. Finally, after nearly 3 years, the one thousand and one nights of the title, the Sultan relents and permanently revokes the death sentence.

This tale is precisely the classic notion of an endless serial, which serialized novels simply attempt to instantiate in a different medium. OP asks for the origin of the marketing concept of publishing a serialized story; a set of novels on a single storyline that will captivate the reader and entice additional sales because the reader simply must find out how the story ends. The earliest origin of that concept I believe to be not the 1001 tales themselves, but the backstory behind them - the story of Scherazade, and the 1001 Tales of the Arabian Nights.

Rabelais wrote five consecutive novels about the giants Gargantua and Pantagruel. This was between 1532 and 1564. A bit later Shakespeare wrote Henry VI part 1, 2 and 3. A thousand years earlier Sophocles wrote Oedipus the King and Oedipus at Colonnos. And some 500 years earlier Homer wrote the Iliad and the Odyssey. All of this is long before James Fenimore Cooper.