32 upvotes
IdeaTaste benchmark checks if LLMs think like researchers
What is IdeaTaste: Do LLMs Share the Research Taste of Human Researchers??
IdeaTaste measures how closely a language model’s paper-selection and problem-picking decisions match those of human scientists. It provides a set of research-topic prompts and compares LLM choices to a crowd-sourced human baseline.
Who it is for
AI researchers can use it to see whether their models have a realistic research intuition before deploying them for literature review or idea generation.