🔍 Read the full analysis: The Referee Problem In An Age Of Cheap AI Production on ThorstenMeyerAI.com
Get school and study supplies delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
A source essay argues that AI is making it far cheaper to produce work than to verify it, citing mathematical manuscripts, software pull requests and contract tasks. The examples point to growing pressure on human reviewers, though several figures come from vendors and the full evidence and methods are not provided in the source material.
An analysis published this week by ThorstenMeyerAI.com argues that AI systems are producing research, code and professional drafts faster than people can verify them, creating a growing human-review bottleneck. The essay points to OpenAI’s mathematical manuscripts, software pull-request data and a legal-workflow evaluation as examples, while some of its figures come from commercial vendors and cannot be independently assessed from the material provided.
The essay says OpenAI’s mathematical research programme generated 722 manuscripts grouped into 372 families after its model was given about 4,000 problems. It reports an average of roughly three hours of compute per result. Some work was formally checked in Lean, a proof assistant, while OpenAI cautioned that results without formal verification could contain issues. The source does not provide the manuscripts themselves or a full account of how the output was selected and assessed.
To illustrate the effort involved in review, the essay contrasts that volume with the response to an earlier result from the programme: a counterexample to an Erdős conjecture that it says was carefully checked by five leading mathematicians. That example illustrates the essay’s central distinction: automated tools can check whether a proof follows from a stated claim, but people still need to assess whether the claim is relevant, correctly framed and meaningful.
In software, the essay cites Faros AI data associating high-AI-adoption periods with 98% more merged pull requests and a 91% rise in review time. It also cites LinearB’s analysis of 8.1 million pull requests across 4,800 organisations: AI-generated changes reportedly waited 4.6 times longer for review to begin and were accepted 32.7% of the time, compared with 84.4% for human-written changes. These are reported findings, not a controlled comparison established by the source material.
The referee shortage: AI made doing cheap and checking expensive
OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.
Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.
Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.
Real progress. Someone still has to find the other 45% before the work can be used.
Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.
A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.
Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.
of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).
of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.
OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.
Budget review hours next to model spend.
Provers, types, tests, policy engines.
Experts only where consequences are high.
Who profits from generation pays for checking.
Keep some production human for learners.
The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.
Review Capacity Is Becoming a Constraint
If the pattern described in the essay holds, organisations may be able to generate more work than they can responsibly use. Review capacity then becomes a practical limit on the value of AI: faster drafting does not automatically produce faster decisions, safer software or dependable research. Work can pile up awaiting inspection, or be approved with less scrutiny than its consequences warrant.
The essay identifies three possible responses: reviewers may approve work too quickly, give AI-generated submissions lower priority, or leave producers to decide which outputs deserve attention. Each carries a cost. Rubber-stamping can let errors through; suspicion can delay sound work; and relying on the producer’s own selection leaves less independent scrutiny. The source presents these as risks and examples, not as a measured account of how often each occurs across industries.
There is also a workforce concern. The essay argues that junior employees often develop judgement by doing the drafting, coding and proof work that AI tools can now take on. If entry-level workers mainly review machine-generated material, employers could weaken the experience pipeline that produces senior reviewers. That is a concern about future training and staffing, rather than a demonstrated outcome in the data supplied.
As an affiliate, we earn on qualifying purchases.
Evidence Across Three Workflows
The analysis links three areas rather than reporting a single new scientific finding. In mathematics, it frames the issue as a gap between formal verification and expert adjudication. A proof checker can test whether a formal proof establishes its stated theorem; it cannot, by itself, decide whether the theorem addresses the intended question or merits attention.
For software, the source cites Faros AI and LinearB, both of which sell products related to software development or code review. That commercial interest does not invalidate their findings, but it is a reason to examine methods, definitions and comparison groups before treating the figures as representative of the whole industry. The essay also refers to a peer-reviewed 2026 study reporting that 61% of AI-agent pull requests received no human review before being merged or closed. The supplied material does not identify the study’s authors, sample or publication venue.
For professional work, the essay describes an OpenAI partnership with contract-software company Ironclad. It says GPT-6 Astra met an average of 55% of evaluation criteria across 11 contracting tasks, an improvement over a previous model. The source does not give the earlier score, explain the evaluation design or establish how well those tasks represent real legal work. The result should be read as a reported benchmark, not proof that the model is ready to handle contracts without oversight.
software pull request review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Broad Is the Review Gap?
The supplied source does not include links, underlying datasets or complete methods for the cited studies and vendor analyses. It is therefore unclear how well their samples represent the wider software sector, whether the reported changes result from AI adoption itself, and how review time and acceptance were defined. The stated percentages should retain their original comparisons; they do not establish a universal effect for every team.
Several other details remain open. The source does not say how many of the 722 mathematical manuscripts were independently reviewed, how many were formally verified, or what happened to the remaining outputs. It also does not provide enough information to judge the Ironclad evaluation’s design or real-world performance. The essay’s claim that review costs may rise is a broader interpretation; the examples show reported patterns, not a single comprehensive measure of verification capacity across professions.
formal verification tools for mathematics
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Methods and Oversight to Watch
The next useful evidence would include publication of the mathematical work and a clear accounting of which results received formal checks and independent expert review. For software, further reporting should show study methods, comparison periods and review outcomes across organisations, including whether AI-generated changes differ in complexity or risk. Those details would help distinguish a general review bottleneck from effects specific to the datasets cited.
For contract work, the evaluation criteria and task-level results would clarify what the reported 55% score means in practice. Meanwhile, organisations adopting AI can track how review queues, acceptance rates and defect rates change, and whether junior staff still get opportunities to build drafting and coding skills. The central question is not only how much AI can produce, but how institutions will verify that output and develop people able to take responsibility for it.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main development described?
A recent analysis argues that AI is lowering the cost and time required to produce work faster than it is lowering the effort needed for people to verify that work.
Did OpenAI publish 722 mathematical manuscripts?
The supplied source says OpenAI’s programme produced 722 manuscripts from about 4,000 problems. It also says some results were formally checked in Lean and warns that unformalized results could have issues. The material provided does not independently document the manuscripts or their review status.
What do the software figures show?
The cited analyses report longer review waits or times for AI-generated work, alongside increased output in some teams. The figures come partly from commercial vendors, and the supplied source does not provide enough methodological detail to establish that they apply across software organisations.
Can AI check its own output?
Automated tools can check specific properties, such as whether a formal proof follows from a stated theorem or whether code passes defined tests. Those checks do not necessarily establish that the theorem, tests or task specification capture the intended problem. Human review remains relevant to those judgements and to accountability.
What remains uncertain?
The source does not provide full methods for the cited studies, the status of every mathematical manuscript, or enough detail to judge the contract-task evaluation. It also does not establish how widespread the reported review pressures are across industries.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
