TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A study from Ai2 tests whether large language models can reliably judge the novelty of scientific research ideas. Across six models, small changes to evaluation prompts produced large shifts in performance, while retrieval and extra reasoning did not consistently help.
Researchers at Ai2 have released a study and benchmark testing whether large language models can reliably judge the novelty of scientific research ideas. In experiments with six LLM judges, changing only a few words in a prompt shifted accuracy on a pairwise novelty task by more than 50 percentage points, raising questions about using these systems to screen ideas for originality.
The study, titled Old Ideas, Novel Problems, evaluates models including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Ai2 describes a novelty judge as a model asked to assess whether an idea is new relative to prior work, either by rating it or comparing it with another idea. The report says seemingly small differences in evaluation setup can lead to sharply different judgments of the same ideas.
To build its first test set, the team mined ICLR 2026 peer reviews for explicit comments about originality rather than treating acceptance as evidence of novelty. Its high-novelty pool contains 154 ideas from accepted papers in the top-rated 10% of their research area, where reviewers praised novelty and none disputed it. The lower-novelty pool contains 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none praised novelty.
The researchers also created a Human+Generated setting, pairing the 154 human ideas with 154 ideas from a basic LLM generator. The team labels the generated ideas as weakly labeled as less novel, not as individually verified cases. Ai2 reports that retrieval and additional reasoning did not consistently improve judgments, and that two purpose-built novelty-evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Novelty Checks Can Mislead
Novelty assessments can influence which research directions receive attention, how automated discovery systems are evaluated, and whether researchers trust AI-generated proposals. If a model’s verdict changes substantially when a prompt is lightly reworded, teams may be measuring the wording of an evaluation as much as the originality of an idea.
The results do not show that every LLM novelty judgment is wrong. They show that, in these tests, performance was sensitive to evaluation choices and was not consistently improved by adding literature retrieval or more reasoning. That makes it difficult to treat a single model score as dependable evidence that an idea is new—or not new.
Ai2 says it is releasing the benchmark, data, code and an automated pipeline so other researchers can test novelty judges and create evaluation sets. The materials offer a way to study the problem, but a benchmark cannot by itself establish that a system will work reliably across fields, datasets or real research workflows.
Products Worth Considering
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
Logitech MX Keys S Wireless Keyboard, Graphite
Amazon Kindle Scribe Colorsoft 64GB (1st Generation)
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Test Sets
Novelty judges are often built from a combination of model choice, prompt, evaluation format, reasoning budget and sometimes a literature-retrieval system. The report says there is no standard way to construct them. It also argues that such systems are often validated on human-written papers even when their intended use is judging AI-generated ideas.
Ai2’s Human-Only test uses reviewer comments as evidence about originality, while keeping novelty distinct from acceptance: papers can be accepted for strengths beyond originality, and rejected papers can still contain original work. In a check of the review-mining process, an annotator agreed with the extracted novelty-signal direction in 47 of 50 cases; the report says none of the disagreements changed whether a reviewer was counted as praising or disputing novelty.
For the generated-idea comparison, the generator received a research area and an instruction to propose a novel idea, but had no literature search, tools, iteration or feedback. In a separate expert review of 30 pairs, selected with extra weight given to cases the strongest judge got wrong, the expert preferred the human idea in all 30 and judged none of the generated ideas novel. Ai2 also rewrote ideas into a common structure to check whether writing style explained model judgments.
““seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2, in its report
Limits of the Novelty Labels
The benchmark’s labels are evidence-based but not a definitive measure of novelty. The high- and low-novelty groups were selected using reviewer judgments and paper rankings, which may not capture every aspect of originality. The Human+Generated comparisons rely on weak labels: the researchers assumed the basic generator’s ideas were less novel than the paired human ideas, but did not verify every generated idea independently.
The report’s supplied description does not specify the exact prompt changes behind the largest accuracy shift or provide the full set of model-by-model scores. It also does not establish how results would vary across other research disciplines, different generators, or evaluation settings beyond those tested. The findings should therefore be read as evidence of instability in this benchmark, not a universal ranking of models or proof that LLMs cannot help assess originality.
Testing Beyond This Benchmark
Ai2 has published the paper, benchmark data, code and pipeline for researchers to examine and use. Further evaluation can test whether the reported prompt sensitivity appears in other fields and with other sets of human and AI-generated ideas, and whether revised judging methods reduce disagreement without relying on favorable wording.
For now, the report supports caution rather than a blanket verdict: LLMs can be tested as novelty judges, but their results in these experiments were sensitive to setup. Whether the tools can become dependable aids for research screening remains an open question.
Key Questions
What did the Ai2 study test?
It tested whether six LLMs could assess the novelty of scientific research ideas, including through comparisons between pairs of ideas and absolute ratings.
How much did prompt wording affect results?
Ai2 reports that changing a few words shifted accuracy by more than 50 percentage points in identifying which of two ideas was more novel. The supplied report summary does not give the exact prompt variants or full model-level scores.
Were the AI-generated ideas confirmed to be less novel?
No. The researchers treated them as weakly labeled as less novel than paired human ideas; that assumption was not verified for every generated idea. An expert separately reviewed 30 selected pairs and preferred the human idea in all of them.
Does the study show LLMs cannot judge research novelty?
No. It reports that judgments in the tested setups were highly sensitive to evaluation choices. It does not establish that every model or method fails in all fields or research settings.
What has Ai2 released?
Ai2 says it has published the study, benchmark, data, code and an automated pipeline intended to help researchers test novelty judges and build further evaluation sets.
Source: huggingface.co
