LLM "hallucinations" persist largely because of how we evaluate models, not just

Artificial Intelligence

LLM "hallucinations" persist largely because of how we evaluate models, not just how we train them Adam Kalai (OpenAI), Ofir Nachum (OpenAI), Santosh Vempala (Georgia Institute of Technology), Edwin Zhang (OpenAI and co-founder with Henry Gasztowtt of the buzzy mysterious g̶o̶l̶d̶ ̶t̶r̶a̶d̶e̶r̶ multi-agent AI Isara) have an "accelerated publication" manuscript (read: unedited, unformatted) in Nature today. 1️⃣ They claim that pretraining creates statistical pressure toward hallucination. Using learning theory, they show that next-word prediction, even on perfect, error-free training data, will produce errors on facts that appear rarely (e.g., a birthday mentioned once in the corpus). The generative error rate is at least roughly twice the rate at which the model would misclassify outputs as valid/invalid. Crucially, this means hallucination isn't just "garbage in, garbage out", it's baked into the objective when facts lack repeated support. 2️⃣ Standard evaluations actively reward guessing over honest uncertainty. Most popular benchmarks (GPQA, MMLU-Pro, SWE-bench, HLE, etc.) use binary grading where abstaining ("I don't know") scores the same as being wrong. Under this scoring, guessing is a dominant strategy: a model that always confidently guesses beats a more reliable model that abstains when uncertain. So even when alignment techniques successfully reduce hallucination, those models lose on leaderboards, which discourages adoption. Their solution will sound obvious to anyone who thought their 3rd-grade teacher graded them too harshly: correct = 1, incorrect = −1, abstaining = 0. Just as students adapt to scoring systems, LLMs adapt and hallucinate less without loss of accuracy. An old elementary school scoring system gets you published in Nature ( 🤓 although it requires advanced math to prove it).