AI-generated vs human-written manuscript assessments: a comparison

You’ve written your novel, and now it’s time to have it edited. When you’ve laboured so long over a text, it can be hard to evaluate it objectively yourself. Another set of eyes can help identify plot holes, weak characters, poor pacing, and errors in spelling, grammar and punctuation. Qualified editors do all this and more; their job is to help authors get their intended message across to readers as accurately, clearly and stylishly as possible. They are trained in how to notice and identify problems in a text and advise authors on how to resolve them. But now AI has entered the arena, and many authors are asking the question: why pay an editor when I can use AI to edit my manuscript for free?

It's a fair question. No one wants to shell out money unnecessarily. There are many arguments against using AI in creating your novel (data centres’ huge use of energy and water; tech companies’ theft of authors’ work to train its LLMs; flooding of the market by AI slop, making it harder for books written by humans to be noticed; devaluation of human creativity, and more), but the one I want to explore in this article is the quality of work an AI ‘editor’ produces.

The word ‘editing’ encompasses a whole range of tasks, so for the purpose of this exercise, I will focus on one: the manuscript assessment. A manuscript assessment is a detailed report, written by a professional editor, on ‘big picture’ elements of a manuscript. It might include comments on such issues as the plot, setting, pacing, characters, authenticity of dialogue, narrative flow, clarity, suitability of the language for the target audience, consistency of the voice, cultural sensitivity issues and so on. When I do a manuscript assessment, I also use Word’s commenting tool to point out some examples of weaknesses and strengths in the manuscript itself.

A little while ago an author asked me to do a manuscript assessment of his novel. He had previously hired another ‘editor’ to do an assessment and had been shocked and dismayed when the report he received (and paid for) turned out to have been generated by AI. The author had not authorised the use of AI, which raises a very serious ethical issue, but that’s not the matter I want to discuss here. I was outraged on the author’s behalf, but also curious about the quality of that AI-generated report: how valid was it? How useful? Did it offer anything that a human editor might not? And, most importantly for an editor concerned about the future of her career, what could a human editor offer that an AI-generated report could not?
I wrote an editorial report of my own without seeing this AI-generated report and sent it off to the author. Then I asked him if I could read the original report, hoping to find answers to my questions. It was a fascinating exercise. Here’s what I found.

• The AI-generated report did have some merit. It identified some of the same strengths and weaknesses in the manuscript that I had found and reported on.
• The AI report showed all the tells of AI writing – in particular, lots of short, choppy, overly dramatic sentences that were perhaps more suited to advertising or poor fiction than a professional critique.
• The advice it gave was very vague. In the way that horoscopes are ambiguous enough to be open to interpretation, you could apply this report’s advice to any manuscript and it would fit. For example, ‘When you’re revising, ask yourself: does each plot point change the circumstances, develop the characters, heighten the danger or reveal a truth?’ (These are not the actual words, but they were along those lines.) And sure, ask yourself those questions – in writing any novel. You could get this advice from a book on creative writing; when you pay for an editor’s services, you expect them to point out how this advice applies to your manuscript in particular.
• It was very light on detail and overall quite brief (four pages compared with my ten pages).
• The report gave no hints as to the editor’s personality. Not that a report needs to show the editor’s personality, but the lack of it, I think, demonstrates the dearth of real engagement and depth of thought that is inevitable with an AI critique. Can AI have an emotional response to your novel? Not a real one. And surely most authors hope to elicit some kind of emotional response from their readers. How can AI gauge whether your work is effective in that regard if it can’t feel? How can you believe it when a machine tells you that your book made it laugh or cry?
• Crucially, the report didn’t flag any legal issues. Editors are not lawyers; we are not qualified to give legal advice. But we might notice some issues that could cause problems for the author – like, among other things, possible copyright breaches or potential for defamation. I had recommended that the author seek legal advice on a few elements of the text. The AI-generated report did not.

• The ‘editor’ had provided an AI-generated line edit at the same time as the assessment, which makes no sense whatsoever. An editorial assessment might lead to major changes when the author uses it to aid the next revision. It could result in whole chapters being deleted or moved, new scenes added, characters deleted or added or merged, endings revised. Line editing before that is done is pointless, for it will only need to be done again on the next version. Besides, the AI-generated line edit that accompanied this AI-generated report was bizarre in the extreme; it included insertions of extra text that were unnecessary and unasked for and made no sense at all. The fact that the ‘editor’ sent an assessment and a line edit at the same time (and that the line edit was such nonsense) demonstrated clearly to me that very little real thought had gone into the exercise.

Yes, editors can be expensive. Yes, editors (even expensive ones) can make mistakes. Yes, there is a certain amount (even a lot) of subjectivity in any manuscript assessment and edit; no two editors will produce the same report. All this is true. From this exercise, however, I concluded that the AI-generated report did not offer the author anything more than my human-generated report did, and that it offered significantly less in important ways. It was too brief and vague to be useful and, crucially, it failed to flag potential legal issues. If this sample of one is typical of AI-generated editorial reports, the latter defect could have serious – and expensive – ramifications for authors.
​
Using AI to edit your manuscript might seem a cheaper option than hiring a human editor, but as far as quality goes, the result doesn’t come close.