And the point is that troff is actually an excellent tool with very beautiful typesetting. The fact that Knuth didn't consider it good enough says a lot about him and his ambitious level of exigence.
Actually, troff was simply not an option for Knuth, for least a couple of reasons.
For one thing, at the time, Unix and all its associated tools were really not prevalent outside of Bell Labs and a few places — troff would not have been available to run on the computer systems Knuth had available to him in 1977. (The ones at SAIL, which were PDP-10 machines running either TOPS-10 or WAITS, I forget what I read.) He didn't have the source code of troff either; all he had was published descriptions: and those he did use, e.g. the paper by Kernighan and Cherry describing `eqn` came out in 1975 and he used ideas from it while designing TeX's syntax for math.
For another thing, troff was designed to control line printers and other such devices that were typically attached to computers; it simply wasn't designed to drive a digital typesetter of the kind Knuth was thinking of (Alphatype CRS). So he'd need a new program anyway. In fact, he says that until he saw Patrick Winston's book on AI in 1977 (which was one of the early books produced on a digital typesetter, i.e. where the actual technology for getting ink onto the page was based on 0s and 1s, instead of being based on either pieces of lead (hot metal typesetting) or light and lenses (phototypesetting)), he had simply not associated output from computers with “real books”. Things like troff were designed to control the simplistic devices that came with computers, that could move by discrete (large!) amounts on the paper (e.g. some could move by full “lines” only), and strike down specific symbols (from a limited font repertoire that came with the device) onto the page. This wouldn't have seemed relevant to Knuth, who primarily wanted to create his own digital fonts (as he did with METAFONT), and TeX was among other things simply a way to use those fonts.
So, regardless of whether one considers troff to have very beautiful typesetting or not, it simply wouldn't have seemed applicable to the problem at the time, to be rejected as "not good enough" or not. Now of course with output going to generic page-description formats like PDF (even TeX's DVI was not Knuth's idea—he was thinking TeX would just write code for the device directly—and he had to be convinced to do it by Fuchs: https://tug.org/interviews/fuchs.html), and high-resolution printers (fairly high DPI, even if not 5000+ as with the CRS then) available to typical consumers, we don't appreciate all this. :-)
I think your main point is correct, that phototypesetting at the time was really bad and that's what drove Knuth to write the tools. (He had been happy to publish with Addison Wesley because of the quality of their typesetting, and now if they were going to move to such an inferior process, it was very upsetting.) The details are off though: phototypesetting is precisely not digital typesetting — in phototypesetting, the decision of where ink goes on the page is controlled via light and lenses and a fixed set of fonts that have been made (physically) for them; it is not controlled by arbitrary 0s and 1s, which was the appeal of digital typesetting. About troff, see my other comment in this thread.
