How I Compared 71 AI Model Configurations On My News Pipeline

🔍 Read the full analysis: How I Compared 71 AI Model Configurations On My News Pipeline on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on his news-writing pipeline and chose GPT-6 Luna for production, despite GPT-6 Astra scoring higher, because Luna cost far less per article. The test also led him to remove two fallback models after they produced fabricated quotes or unusable output, highlighting risks that general model rankings may not reveal.

Meyer, writing on ThorstenMeyerAI.com on October 13, 2026, said the comparison ran from Sept. 8 through Oct. 13 and used his production prompt with real news sources. The configurations covered API, subscription and free-tier models, as well as open-weight models run on his own hardware. Each source was turned into an article in English and German.

The initial test used two sources: a 6,340-character product filing and a 619-character political news item. Meyer said the shorter source better reflected much of his pipeline. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored articles from one to five on source fidelity, fabrication, epistemic clarity, news value, structure and polish, with German language quality also assessed. Stronger configurations then underwent a robustness run of 10 additional articles, mostly based on short sources.

Meyer reported that GPT-6 Astra scored 4.00 out of 5 in each language in the initial test, with source fidelity of 5.00, and averaged 3.93 across 12 articles in the robustness run. He put its cost at about $0.12 per article and its generation time at around 50 seconds. GPT-6 Luna scored 3.67 in English and 3.68 in German in the initial test; its robustness average was 3.67. Meyer said Luna cost roughly $0.0012 per article in the comparison and took about 17 seconds. In production during the preceding week, he reported a cost of about $0.0013 per article at a median 17 seconds.

Why Luna Made the Production Cut

The results illustrate a practical distinction between highest measured quality and a model suitable for routine production. Meyer judged Astra’s advantage meaningful in the robustness test, but said its roughly 100-fold higher cost made it too expensive for his news workflow. Luna scored lower but delivered consistent, valid articles at a much lower reported cost.

For publishers and other teams using generative AI at volume, the findings point to the value of testing models against their own sources, prompts and editorial standards. A broad benchmark or a model’s reputation in another task may not predict whether it will preserve attribution, avoid invention or return output a production system can use. Meyer’s results are one operator’s test, not a universal ranking, but they show why quality, latency, cost and failure behavior all matter in a deployment decision.

The reported fallback failures also carry editorial implications. A backup model may be called only when the primary system fails, but its output can still reach readers if the pipeline accepts it. Testing the full route, including fallbacks and output repair, can expose risks that a comparison of primary models alone would miss.

Testing Models on Real News Sources

Meyer said the test was designed to resemble his day-to-day workflow rather than reproduce a general intelligence leaderboard. Each model used the same production prompt and source material, and an article counted only if it produced valid output after the repair step used in production. The evaluation therefore included both writing quality and whether the pipeline could use the result.

He cautioned that the first comparison involved only two source articles and set a rule that differences below about 0.3 points could be noise at that stage. He treated the larger robustness run as the more dependable ranking and reported 95% bootstrap intervals for those results. In that run, Astra’s interval was 3.83–4.02 and Luna’s was 3.59–3.76, which he said did not overlap.

Meyer also tested GPT-6 Luna at six reasoning-effort settings. He reported no consistent quality gain from increasing effort: the highest setting cost about 12 times more and took about 12 times as long as the no-effort production setting. He said he kept the latter for production. These measurements and decisions apply to his tested task and setup; they do not establish how the models perform on other types of news work.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, writing on ThorstenMeyerAI.com

Limits of the Reported Comparison

The results come from one publisher’s pipeline, one production prompt and two source articles in the initial round. The later robustness test adds 10 articles, mostly based on short sources, but the report does not establish how the rankings would change for other news topics, languages, prompts, publication standards or model versions. The scores were also produced by two AI judges rather than a human editorial panel, and the source material does not describe independent replication.

Meyer reported that both judges saw the original source and scored several editorial dimensions, but the published summary does not provide all underlying article-level scores or explain how disagreements between judges were resolved. It is also not clear from the report whether the cost figures include every production expense beyond model use, or how the results might shift as model prices and services change.

The report says the two fallback models were removed on October 7, but its description of the replacement route ends with an incomplete statement that GPT-6 Luna now runs first, followed by GPT-6 Luna. The identity of the intended second-choice model is therefore unclear from the supplied account.

Production Use and Further Testing

Meyer said GPT-6 Luna was the production model at the time of publication and that he had decided on Oct. 7 that Astra was too expensive for news. He also said the two fallback models were removed from the route on that date. The next practical test will be whether Luna continues to meet his quality and validity standards across ongoing production articles, beyond the limited test set described in the report.

The report does not announce a date for another benchmark or a replacement fallback configuration. Meyer’s next steps were not specified in the source material. Readers evaluating similar systems would need fuller details about the final failover route and additional results across a wider range of sources to judge how well these findings apply elsewhere.

Key Questions

Which model scored highest in the test?

Meyer reported that GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test. Its average across 12 articles in the robustness run was 3.93.

Why did Meyer choose GPT-6 Luna for production?

Meyer said Luna produced consistent, valid articles at a much lower reported cost. He put its cost at about $0.0012 per article in the comparison, against about $0.12 for Astra, and said Luna took about 17 seconds rather than around 50.

Did higher reasoning effort improve Luna’s articles?

Meyer reported no reliable improvement across the six effort settings. The maximum setting cost and took about 12 times as much as the no-effort setting, while the score differences were within the test’s stated noise range.

What problems did the fallback models produce?

Meyer said GPT-4.1 nano received low source-fidelity scores and generated invented quotes, while locally run gpt-oss-120b produced no valid articles out of four in its test. He said both were removed from the news route on Oct. 7.

Do these results show which model is best for all newsrooms?

No. The comparison used Meyer’s own prompt, pipeline and test sources. The report does not establish that the same rankings apply to other publishers, topics, languages or model versions.

Source: ThorstenMeyerAI.com