How were books published in Ancient Greece?

Alex has given a good general answer. Let me add a detail:

The ancient Jews had a simple system for mass-producing copies of Scripture: Get a room-full of scribes. One stands in front with the book to be copied. The rest sit at desks with blank paper. The guy at the front reads slowly, while the rest write down what he says. Then they had various schemes to check that a copy was accurate. Like always write the same number of letters per line. Then when you get to the end, if the original ends on, say, page 32, line 20, and that line has 15 letters, then if your copy does not end on page 32, line 20, with 15 letters, you must have added or dropped letters somewhere. They'd also spot check places in the middle.

I'm not sure if this technique was used for books other than Hebrew Scripture, but it seems obvious enough that it could have been. Of course this assumes there's a large enough demand for copies to justify getting a bunch of people.

I can only offer fragmentary facts I have read but cannot cite sources.

I was told by university lecturer years ago that there survives an Ancient Greek 'history of philosophers' that mentions one philosopher of that time of whom the most interesting thing to be said about him was that he owned the complete works of Plato, indicating that this was very unusual even for philosophers. A lot of the time people learned more by 'talking to someone who had talked to Socrates'.

In Latin (hence not strictly Greek, but Greco-Roman culture was closely intertwined) there is a word 'librarius' which can be translated 'bookseller', although I have read that the librarius's profession involved copying out the books as well as selling them. This suggests there were businesses that specialised in disseminating books for profit, although as other have said, there was no copyright law so authors did not have the same kind of connection to them that modern authors do to their publishers.

There was also a custom among educated Romans that if they wrote a book they invited people they knew to attend a reading of it. The Roman historian Tacitus tried to end each chapter with an epigram, apparently to signal to the audience at the reading when to applaud. Other Roman writers complained about how boring the reading of a book by its author could be. One Roman money lender whose hobby was writing histories was said to be notorious for compelling those unfortunate enough to be in debt to him to attend his readings of his histories.

No, the original author did not get royalties. In fact, often times original authors of works would not be known, or people would write works and attribute them to more famous authors in an effort to get them more widely distributed (up to half of Paul's Epistles are thought to have been authored this way).

Since there was no printing, and most people were illiterate, there was essentially a small professional class of literates who could perform duties like taking dictation (like modern court reporters), writing letters (like 20th century telegraph operators), or copying existing works (like a really really slow and inaccurate photocopier). Copying large works was a very labor-intensive job, and since not just anyone could do it, this made books (scrolls) really expensive to distribute. Just about all the money involved would go to the scribe.

Of course if the source work were in someone else's personal possession, you have to make arrangements with them for access. If you both are rich folk with libraries, that might just be tit-for-tat access. After all, he needs to build his library in the same way.

The concept of copyright originated after the printing press, and was meant to be an industrial regulation (since copying was now cheap, but the presses were expensive). The idea was that the local ruler could reserve for an individual person the exclusive right (eg: monopoly) to copy particular lucrative works. This monopoly was at first given not to authors, but to favorites of the ruler. It was the British at the beginning of the 18th century that first legally established the concept of a limited-time copyright for publishers (not authors) as an encouragement for the production of more works. In the USA in the 20th century this slowly evolved into a near-perpetual monopoly to the publisher, with the original goal of producing more works merely an afterthought.

In the modern digital era copying of works no longer requires an industrial-scale investment in equipment. In fact, it requires a large-scale effort of hardware and software to prevent it from being a trivial operation. So there have been some attempts to rethink the value and use of copyright. Probably off-topic to get into that here, but if you are interested I'd suggest reading some of the GNU Philosophy documents, and reading up on the Free Software movement, and the various "Pirate" political parties.