Was Donald Knuth the first person to typeset a book using a computer?

No, Knuth was not the first person to typeset a book "with a computer".

The TEX project was started in 1978 by Don­ald E. Knuth, while re­vis­ing the sec­ond vol­ume of his Art of Com­puter Pro­gram­ming. When he got the galleys back, he saw that the pub­lisher had switched to a new dig­i­tal type­set­ting sys­tem and was shocked at the poor qual­ity.

What are TEX and its friends? (CTAN Comprehensive T_EX Archive Network)

And this is indeed a humble beginning, as it took quite a while to fine-tune TeX78 until realising that so many design mistakes were made that a fresh start was called for:

14 Mar 1978
 ...
 # 4:30am, \TeX's first page is successfully output!
 ...
* 26 Mar 1978
...
# Finally the entire test program was \TeX ed. Happy Easter! Six hours today.

From http://www.ctan.org/tex-archive/systems/knuth/dist/errata#errorlog.tex

That manual was published in September 1978. As Knuth wrote the user manual does not seem to count as a real book and he mentions in the article a 28-page book that appeared end of 1978 as the first book published by TeX.

Of course, user manuals don't count as real milestones in publishing. I like to think that the first real book to be printed with TEX was a 28-page keepsake that was made for my wife's relatives at Christmastime, 1978. This book included eighteen original linoleum block illustrations, into which we pasted XGP-produced text set in a special 14-point extended variant of the prototype Computer Modern font. In order to compensate for the XGP's limited resolution, we prepared magnified copy and the printer reduced it to 70%; the effective resolution was therefore about 286 pixels/inch. The title, opening pages, and colophon are illustrated here (reduced another 65% from the published size). About 100 copies were printed, of which roughly 25 were sold and the remaining 75 were given as gifts. A complete library citation for this book would read as follows: "Lena Bernice: Her Christmas in Wood County, 1895. By Elizabeth Ann James, with illustrations by Jill Carter Knuth. Columbus, Ohio: Rainshine Press, 1978."

Donald E. Knuth: "TeX Incunabula", TUGboat, Volume 5, No. 1, 1983 (PDF) Illustration in there of the first pages)

It only got real, arguably, then with Donald E. Knuth: "The TEXbook", Addison Wesley, 1984, using TEX Version 3.0:

Gentle Reader: This is a handbook about T_EX, a new typesetting system intended for the creation of beautiful books—and especially for books that contain a lot of mathematics.
(From the preface, emphasis added)

In fact Knuth was inspired to develop his system after seeing another book:

He had just received his first samples from the new typesetting system of the publisher's, and its quality was so far below that of the first edition of Volume 2 that he couldn't stand it. Around the same time, he saw a new book (Artificial Intelligence, by Patrick Winston) that had been produced digitally, and ultimately realized that typesetting meant arranging 0's and 1's (ink and no ink) in the proper pattern, and said (approximately), "As a computer scientist, I really identify with patterns of 0's and 1's; I ought to be able to do something about this", so he set out to learn what were the traditional rules for typesetting math, what constituted good typography, and (because the fonts of symbols that he needed really didn't exist) as much as he could about type design.

Curiously, while the first edition of the book by Winston was the inspiration for TeX, already the second version came out in 1984 using TeX! Comparison to earlier methods used at the MIT Press, the progress is obviuos.

And the same page lists the real advantages TeX can claim for itself:

As it turned out, TeX was still a lot closer to a research project than to an industrial strength product, but there were certain attractive features:

• it was intended to be used directly by authors (and their secretaries) who are the ones who really know what they are writing about;
• it came from an academic source, and was intended to be available for no monetary fee (nobody said anything about how much support it was going to need);
• as things developed, it became available on just about any computer and operating system, and was designed specifically so that input files (files containing markup instructions; this is not a WYSIWYG system) would be portable, and would generate the same output on any system on which they were processed—same hyphenations, same line breaks, same page breaks, same everything.
• other programs available at the time for mathematical composition were:
  • proprietary;
  • very expensive
  • often limited to specific hardware,
  • if WYSIWYG, the same expression in two places in the same document might very well not look the same, never mind look the same if processed on two different systems.

Seeing that a very early contender for the title "first" would be Probably the First Book Typeset by Computer 10/6/1963 we might get quite a few other candidates to consider. But that depends on the exact criteria used. From first book that had 'a computer used' during any part of its production run, to 'first book that was set entirely on an online computer typesetting programme.

Big iron typesetting aids running on mainframes and using punch cards to drive phototypesetting came into use during the 50s and were instrumental in 60s magazine production. So the criteria to count would have to be specified to avoid producing an overly long list of increments here.

One of the most important would probably be:

1963 Compugraphic introduces their first typesetting computers, the Linasec I and II. These typesetting computers are used to prepare punched tape to drive the Linotype typesetting machines used in the newspaper industry. The paper tape is produced on a Teletype BRPE punch at the rate of about a line a second, hence the name ‘Linasec’. The design and initial manufacturing of the machines are outsourced to Wang Laboratories, another pioneer in the computer history.

If these second generation phototypesetters with patents dating from 1944 onwards count as computing devices, then 1952 saw the first book published with this invention: The Wonderful World of Insects.

Comparing that to TYPSET and RUNOFF from 1964, the ancesors of roff, troff, nroff, groff etc, gives the impression that UNIX-like systems might have had a headstart there and TeX might compare quite poorly in usability to GUI based DTP systems that also went for sale during the mid80s, but the real kicker was the mathematical formula typesetting algorithms and the line break algorithms in LaTeX, still not surpassed in most recent modern word-processors and very few DTP-programs.

Allow me a slight advertis*m*nt for a similar system of about the same vintage less inclined for mathematics but much more suited in design principles for historians, linguists etc, and also from 1978: TUSTEP (Homepage, example for a critical edition of Ulysses, in 1979, precursor use of computer aided typesetting and preparation for photosetting: 1973). For type-setting eg critical apparatuses and the like this was always superior to TeX. But the learning curve is equally unfriendly. First modular ancestors for this system would have to count from 1966 onwards.

Two other important steps have to be included:

Andrew W. Bingham: "Photon: Printing Revolution – Photographic Typesetting May Replace Linotype", The Harvard Crimson, 1956:

A Computing Machine?

At first glance, the Photon machine looke like a compact IBM-type computing machine. It has only three major components: a standard Underwood electric typewriter, a telephone relay system, and a photographing unit. Its basic difference from the conventional method of hot-metal type-setting are two. The keyboard of a typecaster is big and complex. Photon uses the keyboard of a standard electric typewriter. And secondly, the end product is different. The old machine casts individual lines of type. Photon, on the other hand, actually sets no type. It simply reproduces, on film, type in any style and size.

By manipulating the appropriate controls located beside the actual keyboard, the Photon operator can pre-set the width of the column and the style and size of the type. Then he can start touch-typing on the keyboard. When he reaches the minimum number of spaces to fill out the desired width, a bell will ring. Another bell will ring upon reaching the maximum number of characters for one line. When this point has been reached, the machine will lock automatically.

And of course the ancestor of editors in that direction

Colossal Typewriter by John McCarthy and Roland Silver was one of the earliest computer text editors. The program ran on the PDP-1 at Bolt, Beranek and Newman (BBN) by December 1960.