AllenAI launches benchmark to test LLM novelty judges
What is Can we trust LLMs to judge whether a research idea is novel??
The release includes a dataset, benchmark, and code for measuring how reliably large language models can decide if a research idea is new. Researchers can also run an automated pipeline to build custom evaluation sets and compare different novelty-judging approaches.
Who it is for
Anyone building AI tools for scientific discovery can use this to verify that their models aren’t just repackaging old work.
How to try it
Open the Hugging Face blog at https://huggingface.co/blog/allenai/novelty-bench