🔍 Read the full analysis: What’s The Best AI Model For A News Pipeline? My Tests Had A Surprise on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Thorsten Meyer reports testing 71 AI model configurations on news-writing tasks from Sept. 8 to Oct. 13, 2026. GPT-6 Astra scored highest, while GPT-6 Luna cost about $0.0012 per article and performed reliably; two fallback models were removed after producing fabricated quotes or unusable output.
Thorsten Meyer says a test of 71 AI model configurations for news writing found that GPT-6 Astra scored highest, but cost about 100 times as much per article as GPT-6 Luna, the model he chose for production. The test also found that two cheaper fallback models produced fabricated quotations or output the pipeline could not use; Meyer says he removed them from the news route on Oct. 7.
Meyer, writing on ThorstenMeyerAI.com on Oct. 13, said he tested the configurations from Sept. 8 through Oct. 13 using his production prompt and real news sources. Each configuration generated English and German articles. Two AI judges scored the results on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with an additional language-quality measure for German.
In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a 5.00 for source fidelity. Across a 12-article robustness run, it scored 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. Meyer put its cost at about $0.12 per article and its generation time at around 50 seconds.
GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test. In the robustness run it scored 3.67, with source fidelity of 4.71 out of 5; Meyer said every Luna article was valid under the pipeline’s production checks. Its reported cost was about $0.0012 per article, with a median generation time of 17 seconds. Meyer said he kept Luna in production because Astra’s higher score did not justify its much greater cost for his workflow.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost, quality and pipeline reliability
The results illustrate why a model leaderboard may not identify the best tool for a specific newsroom task. Meyer’s test measured whether models could produce usable articles from supplied sources, rather than general reasoning or coding ability. In his results, the highest-scoring model was not the production choice: Luna scored lower than Astra but was substantially cheaper and passed the pipeline’s validity check in every test article.
The failures among the fallback models also point to a separate operational risk. A primary model can perform well while a failover route introduces unsupported quotes or breaks the output format. In a news pipeline, those problems can affect factual reliability and publication readiness, not just writing quality. Meyer’s findings suggest that teams should test fallback models against the same source-fidelity and output-validity requirements as their main model.
The results are specific to one publisher’s prompt, source mix, languages, model versions and judging method. They do not establish which model is best for every newsroom. They do, however, provide a concrete example of why evaluation on an organization’s own work can matter more than transferring rankings from unrelated benchmarks.
Popular right now
Shark AI Ultra Robot Vacuum, Black/Silver, AV2511AE
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream
Dyson V8 Cyclone Cordless Vacuum
As an affiliate, we earn on qualifying purchases.
How Meyer tested newsroom models
The evaluation used two real sources: a 6,340-character product filing and a much shorter, 619-character political news item. Meyer described the short-source case as especially representative of his production work. Each model wrote an article in English and German, using the same prompt as the live pipeline.
The two judges were GPT-6.1 Sol, set to high reasoning, and Claude Opus 5.5. Both received the original source and scored the articles from 1 to 5 across several editorial criteria. Meyer also ran the strongest configurations on 10 additional real sources, mostly short ones. He said that, with only two articles in the initial test, score differences under about 0.3 points should be treated as noise; he considered the larger robustness run a more dependable ranking.
Meyer also tested GPT-6 Luna at six reasoning-effort settings. The reported scores varied without a consistent improvement: the maximum setting cost about 1.43 cents per article and took 211 seconds, compared with 0.12 cents and 17 seconds at the production setting, which used no added reasoning effort. Meyer said the differences fell within the test’s noise, so he retained the faster setting.
“A general intelligence score cannot answer the question I actually have to answer: which model should write the news articles my sites publish every day?”
— Thorsten Meyer, ThorstenMeyerAI.com
Limits of the reported results
The results come from one publisher’s workflow, and the article does not establish whether the same rankings would hold for other prompts, publication standards, source types or language pairs. Meyer reported using two AI judges, including Claude Opus 5.5, and did not provide an independent human-rating comparison in the supplied account. The scores should therefore be read as results of this specific evaluation, not as a universal measure of newsroom quality.
The initial comparison relied on two sources per configuration, although Meyer added a 10-article robustness run for the strongest configurations. The supplied material does not list every model configuration or provide article-level scores and judge disagreements. It also does not describe a human review rate or whether any tested article was published without editorial changes. Those details would help readers judge how closely the benchmark reflects publication outcomes.
Meyer says both weak fallbacks were removed on Oct. 7, but the supplied account contains an incomplete description of the replacement route: it says GPT-6 Luna now runs first, followed by “GPT-6 Luna as its” without identifying the subsequent fallback. The current backup model and the pipeline’s present failover behavior are therefore not clear.
Further testing and route details
Meyer’s published account says GPT-6 Luna remains the production model and that GPT-4.1 nano and local gpt-oss-120b were removed from the news route on Oct. 7. It does not specify a date for another test or describe a planned next milestone. The next useful reporting would be confirmation of which model now serves as the fallback, along with updated results from the live route.
For readers evaluating the findings, further evidence could include the full set of tested configurations, the robustness results by model, and a comparison with human editorial review. Because model versions and prices can change, any later recommendation would need to be tied to the versions and costs in effect at the time of testing. Meyer’s current conclusion is narrower: in his measured workflow, Luna met his quality bar at a much lower reported cost than Astra.
Key Questions
Which model scored highest in Meyer’s test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It scored 3.93 across 12 articles in the robustness run.
Why did Meyer use GPT-6 Luna instead?
Meyer reported that Luna scored 3.67 in the robustness run, produced valid output for every tested article and cost about $0.0012 per article. Astra cost about $0.12 per article, roughly 100 times more, according to his figures.
Did higher reasoning effort improve Luna’s articles?
Not consistently in this test. Meyer said score differences across Luna’s six reasoning settings were within the noise, while the maximum setting took 211 seconds and cost about 1.43 cents per article, compared with 17 seconds and 0.12 cents at the production setting.
What happened with the fallback models?
Meyer said GPT-4.1 nano scored 1.62 in English and 1.68 in German, with a source-fidelity score of 1.00, and that judges flagged invented quotations. He reported that local gpt-oss-120b produced no valid articles in four attempts and also invented named quotes. Both were removed from the route on Oct. 7.
Does the test identify the best model for every newsroom?
No. The findings reflect one publisher’s prompt, sources, English and German editions, pipeline checks and AI judges. Other newsrooms would need to test models on their own tasks and editorial requirements.
Source: ThorstenMeyerAI.com
