Is Checking AI’s Work Worth The Effort? What Studies Suggest
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Is Checking AI’s Work Worth The Effort? What Studies Suggest on ThorstenMeyerAI.com

Age 18–24?Offer from Amazon

Prime made for students and young adults

  • Fast, free delivery for dorm and study essentials
  • Prime Video and Amazon Music included
  • Member-only deals
Try Prime for Young Adults Free trial for eligible 18–24 year olds
As an affiliate, we earn on qualifying purchases.

TL;DR

Studies reviewed by cognitive engineer Karen Feigh show that AI’s productivity effects vary by task: it can speed up some writing and consulting work, while a METR trial found experienced developers took longer with AI enabled. The research also points to a verification burden: people spend time checking AI output, and reviewers may miss errors that require understanding task rules.

Studies of AI-assisted work show that faster drafts do not always mean faster completion: in a randomized trial, experienced software developers took 19% longer on assigned tasks when AI tools were available, while separate experiments found gains on some writing and consulting tasks. The findings, discussed in a November 16 essay by Karen Feigh of Georgia Tech, highlight the time and attention workers may need to check AI output before relying on it.

Feigh, a cognitive engineer, calls this checking work the “verification burden.” Her essay in The Conversation is an expert analysis, not a new experiment. It draws on prior research to argue that AI’s apparent speed must be weighed against the effort required to assess whether its answers are accurate and suitable for the task.

The software evidence comes from a METR randomized trial involving 16 experienced open-source developers and 246 tasks in large, mature projects. Developers were randomly assigned tasks with or without AI access. On average, the AI-allowed tasks took 19% longer. The participants had expected AI to reduce their completion time by 24% before the trial; afterwards, they still estimated it had saved them 20%.

METR reported that developers accepted fewer than 44% of AI code suggestions, and 75% said they read every line of generated code. On screen recordings for a subset of 44 issues with valid labels, about 9% of working time went to reviewing and cleaning AI output. These figures apply to the study’s specific developers, projects and tools, including Cursor and Claude models used from February to June 2025; they do not establish a universal cost for AI use.

At a glance
reportWhen: Essay published November 16, 2026; stud…
The developmentA November 16 essay by Georgia Tech’s Karen Feigh brings together studies showing that AI’s time savings can be offset by verification work and vary sharply by task.

AI and work · explainer

The verification loop: where AI’s time savings can go

The draft is fast. Knowing whether it is right is the work that follows.

Time for one task (schematic)

AI draft Checking Fixing Re-checking What a speed metric sees What the work costs

Draft: secondsgood and bad parts in the same confident voice About 9% of the timewent to reviewing and cleaning AI output in METR’s trial with AI allowed Speed is not productivitycount the whole loop, errors included

How the hidden cost builds up

  1. DraftThe AI answers in seconds, fluent and confident“few clues about where scrutiny is needed”
  2. CheckDecide what to check and how, without knowing if or where an error iscognitive load
  3. FixFinding an error does not end the work, it starts a correction cycletime
  4. Re-checkEvery fix needs checking again; when nothing is wrong, the effort feels wastedmotivation
  5. Count it allProductivity is the whole human-AI workflow, not the first draftnet, not gross
The real measure: “whether the entire human-AI workflow produces trustworthy work at a lower total cost”

Source: Karen Feigh, Georgia Tech, The Conversation, Nov 16, 2026 (the three burdens, the quotes); METR, arXiv 2507.09089, p. 20 (time spent reviewing and cleaning AI output). The bars are a schematic, not data.

Graphic: Powered by Thorsten Meyer AI

Why Review Time Changes the Calculation

The findings challenge a common way of judging workplace AI: measuring how quickly it produces a first draft. The relevant measure is total task time, including checking, testing, correcting and integrating the output. Where review is intensive, a quick result may not translate into a faster finished product.

That distinction matters to employers and workers who remain responsible for decisions made with AI assistance. A polished answer can still contain errors, and responsibility does not automatically shift to the system. Feigh describes workers as supervising a fast, capable assistant whose output can be uneven and whose confidence offers limited guidance about where to inspect closely.

The research does not show that AI is unhelpful overall. It indicates that results depend on the task, the user’s expertise and whether the work falls within a tool’s capabilities. Organizations evaluating productivity need to account for accuracy and quality as well as speed, and allow time for people to verify work they are accountable for.

Ninja Luxe Café Premier 3-in-1 Espresso Machine, Drip Coffee & Cold Brew

Ninja Luxe Café Premier 3-in-1 Espresso Machine, Drip Coffee & Cold Brew

Sony BRAVIA 5 85-Inch Mini LED 4K HDR Smart Google TV (K-85XR50)

Sony BRAVIA 5 85-Inch Mini LED 4K HDR Smart Google TV (K-85XR50)

As an affiliate, we earn on qualifying purchases.

Different Tasks, Different Results

Other studies cited in Feigh’s essay found productivity gains in specific settings. A study of mid-level professional writing tasks reported that ChatGPT users completed work 40% faster and produced work rated 18% higher in quality. Those results concern the tasks and participants in that experiment, not every kind of writing or workplace.

In a preregistered experiment conducted with Boston Consulting Group, 758 knowledge workers used AI on consulting tasks. On 18 tasks judged to be within AI’s capabilities, participants using AI completed 12.2% more tasks and finished 25.1% faster on average, with higher-quality solutions. On a complex task outside those capabilities, AI users were 19% less likely to give a correct answer than participants who did not use AI. The contrast suggests that knowing when a task exceeds a tool’s strengths can matter as much as access to the tool.

Checking itself is not always reliable. In an experiment involving 2,784 people reviewing AI-style suggestions, participants usually caught typographical errors, but achieved only 31% accuracy on errors that required understanding the rules of the task. That result warns against assuming that a human review automatically makes an AI-generated answer safe or correct.

“AI’s productivity gains “are often countered by time and mental energy spent determining whether the output is accurate and appropriate.””

— Karen Feigh, cognitive engineer at Georgia Tech, in The Conversation

Where the Evidence Does Not Reach

The studies do not provide one estimate of AI’s productivity effect across all jobs. Their tasks, participants, tools and outcome measures differ, and the developer trial used particular software and models during a defined period in 2025. Its result should not be read as evidence that AI always slows developers, or that future systems will produce the same outcome.

It is also unclear how much the findings generalize to routine workplace use, where workers may have different levels of expertise, training and time pressure. The reported review measures cover limited samples or specific experiments. They do not establish how much checking is enough to catch every consequential error, particularly when the error depends on specialist knowledge.

Feigh’s essay synthesizes prior research rather than reporting a new test of a single verification method. The studies describe risks and trade-offs, but do not establish one review process that will make AI output reliable across tasks.

Measure the Whole Work Process

The practical recommendation emerging from Feigh’s analysis is to evaluate the complete workflow, not just the time required to generate an initial answer. That means tracking review and correction time alongside speed, quality and error rates, while accounting for whether the task is one the AI tool handles well.

Employers also need to give workers who remain accountable for AI-assisted output enough time and training to check it. The cited studies do not prescribe a single standard or indicate a universal threshold for safe review. Further evidence across more occupations and real-world settings will be needed to show when the time saved exceeds the time spent checking—and which review practices reliably catch difficult errors.

Key Questions

Did the METR study find that AI always makes developers slower?

No. It found that 16 experienced open-source developers took 19% longer on 246 tasks when AI tools were allowed in that trial. The result applies to that study setting, not all developers or software tasks.

Why can AI make a task take longer?

Workers may need to read, test, correct or reject generated output. In the METR trial, developers accepted fewer than 44% of suggestions, and a subset of screen recordings showed review and cleanup taking about 9% of working time.

Do studies also show productivity gains from AI?

Yes. Research cited by Feigh found faster completion on certain professional writing and consulting tasks. In the consulting experiment, gains appeared on tasks within AI’s capabilities, while performance was worse on one task outside them.

Can human review catch AI errors reliably?

Not necessarily. In one experiment with 2,784 reviewers, accuracy was 31% for errors that required understanding task rules, even though typos were usually caught. The finding suggests that review quality depends on the kind of error and the reviewer’s understanding.

Source: ThorstenMeyerAI.com

You May Also Like

Introducing Forezai · TradingAgents — a committee of LLMs decides paper-trades

Forezai introduces TradingAgents, a framework where a committee of large language models makes paper-trading decisions, enhancing research capabilities.

The Bubble Question, Disentangled: 1999 vs 2026 Category by Category

A detailed analysis compares the 1999 dotcom bubble with the 2026 AI cycle, examining categories of investments, valuation signals, and future risks.

Painting The Sides Of Railroad Rails White To Reduce Derailment

Rail companies are experimenting with painting rail sides white to improve visibility and reduce derailment risks. The initiative is in early stages and under evaluation.

The AI Exam Nobody Studied For: When the Answer Was Hidden in the Footnotes

Five frontier AIs ran the same company through its worst week. All passed the ethics test — but only two read the footnotes and closed a €55,000 deal.