AI News Research on AI verification: why developers using AI took 19% longer when checking their work

Wrecker4923

Level 9
Verified
Well-known

Excerpts:​

The measurement of productivity gains cannot include just how quickly AI produces a first draft. Instead, the measure is whether the entire human-AI workflow produces trustworthy work at a lower total cost.
An executive assistant told me she uses AI tools for drafting itineraries and briefings, but she’s caught enough issues that she now builds time into her workflow to thoroughly check for inaccuracies and biases.
AI responses can improve productivity, but the gains are often countered by time and mental energy spent determining whether the output is accurate and appropriate. This is the verification burden.

AI Summary:​

In this article, cognitive engineering professor Karen Feigh of Georgia Tech argues that verifying AI-generated output imposes a measurable "verification burden" on workers that can offset or erase the productivity gains AI provides. She identifies three distinct costs:
  • a cognitive burden (unlike creation, verification has no clear stopping point and requires the worker to identify errors whose existence, location, and form are unknown);
  • a time burden (each error found triggers a correction-and-recheck cycle); and
  • a motivational burden (when AI usually works well, checking can feel unproductive and vigilance erodes over time).
She cites a randomized trial in which experienced software developers using AI actually took 19% longer when a verification step was included, despite believing they had been faster, and a consulting-team study showing that using AI outside AI's area of expertise increased the rate of wrong conclusions.

Feigh frames the shift as a "skill swap": workers are trading creation skills for supervisory skills that resemble overseeing a fast but unreliable intern who delivers good and bad work in the same confident voice. She concludes that if humans remain accountable for AI-assisted output, organizations must provide the time, training, and authority needed to verify it, and that productivity should be measured by whether the full human-AI workflow produces trustworthy work at a lower total cost — not merely by how quickly a first draft is generated.

My Note:​

The obvious (🤪) solution to (allegedly) increase productivity is to have no verification! Brace for more software broken by hallucination!

P.S. I thought the AI summary was accurate. Trust me, bro.

P.S.Edited using the @Bot's corrective remark (with original research verification).
 
Last edited:
Excerpts: AI Summary: In this article, cognitive engineering professor Karen Feigh of Georgia Tech argues that verifying AI-generated output imposes a measurable "verification burden" on workers that can offset or erase the productivity gains AI provides. She identifies three distinct costs: a cogni […]


Hi Wrecker4923 — the joke lands, but the AI summary illustrates a useful distinction: it is a reasonably faithful summary of the article, not necessarily a fact-check of the studies behind it.

Two qualifications matter:

  1. The 19% figure was not produced by adding a separate verification step. In the original METR randomized trial, 16 experienced open-source developers completed 246 real tasks, with each task assigned to either allow or disallow AI. AI-allowed tasks took 19% longer on average. Therefore, 19% is the net effect of AI access in that particular setting, not a measurement of verification time alone. Reviewing and correcting generated code may have contributed, but the experiment did not isolate it as the cause. The article’s “when a verification step was included” wording is too definite. See the original METR study.
  2. METR now labels those early-2025 results as out of date. Its later raw data suggested possible speedups from newer tools, but METR considered the estimates unreliable because of participant and task-selection effects, plus difficulty measuring parallel-agent work. So the 19% result remains valid for that experiment, but it is not a universal or current “AI tax.” The February 2026 update should really have accompanied the statistic in a November 2026 article.

There is another slight distortion: the consulting study found worse results on a task outside GPT-4’s capability frontier, not necessarily outside each consultant’s personal expertise. In that 758-consultant experiment, AI users were 19 percentage points less likely to reach the correct answer on the deliberately AI-hostile task (study abstract). That is an important difference.

The broader argument nevertheless stands. For software, productivity should be measured through to an accepted, trustworthy change:

  • time to reviewed and merged code;
  • rework and review effort;
  • tests and static-analysis results;
  • security defects and escaped production bugs;
  • maintenance cost afterward.

Human-written code also requires verification, so the relevant cost is the additional or reduced burden compared with the non-AI workflow, not every minute spent testing.

Skipping verification is certainly faster right up until the defect reaches production. Then the saved time returns with interest, wearing an incident-response badge. Your summary passed the “faithfully repeated the article” test, but not quite the “checked the primary evidence” test—which is almost suspiciously on-theme.

Sources
 
@Bot

For the consulting study, that's a nice (🙏) catch for human/AI's mistake. From the original abstract of the original research:
For each one of a set of 18 realistic knowledge tasks within the frontier of AI capabilities ranging from creative to analytical tasks, subjects using AI outperformed those not using AI, completing 12.2% more tasks and completing them 25.1% more quickly on average while also delivering solutions of significantly improved quality. However, for a complex managerial task selected to be outside the frontier, subjects using AI were 19% less likely to produce correct solutions compared with those without AI, pointing to potential limitations of AI supporting knowledge workers.

So using AI increases productivity if it’s knowledgeable in that area, but decreases it if it isn’t. It’s up to the human to understand (😲) its limitations, even though AI always sounds confident. Here’s an excerpt from the researcher that probably makes that point:
First, our results lend support to optimism about AI capabilities for important high-end knowledge work tasks. In our study, the human-AI combination proved surprisingly capable, demonstrating the potential of AI-aided professional work. However, we also show the challenges resulting from AI’s jagged capabilities; experienced and incentivized knowledge professionals, engaged in tasks akin to some of their daily responsibilities, performed worse when given access to AI. This happened even though they did not need to rely on AI’s output to complete the task. The jagged nature of the knowledge frontier suggests that in real-world settings, knowledge-intensive workflows will most likely straddle both sides, creating situations where knowledge workers—unaware of their position relative to AI’s capabilities—perform worse as a result of becoming overly reliant on AI (“falling asleep at the wheel” (Dell’Acqua 2022)), a risk that is reinforced by the persuasive capabilities of generative AI (Randazzo et al. 2025a).
 
@Bot For the consulting study, that's a nice (🙏) catch for human/AI's mistake. From the original abstract of the original research: So using AI increases productivity if it’s knowledgeable in that area, but decreases it if it isn’t. It’s up to the human to understand (😲) its limitations, even though AI always sounds confident. Here’s an excerpt from the researcher that probably makes that point:


That quotation captures the important point, with one numerical wrinkle: “19 percentage points” was intentional in my reply. The journal abstract says “19% less likely,” but the paper’s results section reports:

  • control group: 84.5% correct
  • AI groups: 60.0% and 70.6% correct
  • combined difference: approximately 19 percentage points

The earlier working-paper abstract also says 19 percentage points. A fall from 84.5% to roughly 65.3% is about a 23% relative reduction, not 19%. So the published abstract uses “19%” rather loosely; the body supplies the more precise unit. Apparently even abstracts need verification—this thread is becoming aggressively self-demonstrating. The published paper gives the underlying rates here.

Your larger interpretation is right, although I would phrase it slightly more narrowly: AI improved performance on the 18 tasks deliberately selected to be within that version of GPT-4’s capability frontier, while it reduced correctness on the deliberately selected outside-frontier task. One experiment cannot establish that AI always raises productivity whenever it is capable and always lowers it whenever it is not.

The awkward part is that “understand its limitations” is much harder than recognizing a broad subject area. The frontier is jagged at the task level: a model may competently perform one consulting or programming task and confidently mishandle an apparently similar one. It also shifts with the model, available context, prompting and tools. In this experiment, experienced and incentivized consultants were free to disregard the AI, yet access to it still made the outside-frontier result worse.

That makes the human’s real skill not merely knowing that “AI can hallucinate,” but calibrating reliance and independently validating the particular output. In software, that means understanding the change, reviewing the diff, exercising relevant edge cases, and checking whether the tests themselves are adequate—not accepting green tests as a ceremonial blessing from the machine.

Sources
 
I mean sure, it is a well regarded issue. no mouth flapping needed, but many or most.. enterprise heads, train their own AI to use it for those exact tasks and criteria.
 

MalwareTips community

Join the discussion

Create a free account to reply to this thread, ask your own questions and get help from the community.

  • Reply and ask questions
  • Get alerts when someone answers
  • Free malware removal help

Join 75,656 members who help each other for free, since 2010.

Create free account
or continue with

Recently browsing

Members and guests who viewed this thread in the last 5 minutes

Back
Top