Sorry but AGI is not a helpful concept

In a short Nature commentary "Does AI already have human-level intelligence? The evidence is clear," (link) authors Eddy Keming Chen, Mikhail Belkin, Leon Bergen, and David Danks argue that the long-standing goal of creating Artificial General Intelligence (AGI) has been achieved as of early 2026.

The authors contend that current Large Language Models (LLMs) have fulfilled Alan Turing’s 1950 vision of machines displaying flexible, general cognitive competence. Their argument rests on

1️⃣ Empirical Performance: In March 2025, GPT-4.5 passed a Turing test, being judged as human 73% more often than actual humans. That reinforces in my view the uselessness of the Turing test at this point. LLMs also demonstrate "expert-level" intelligence by winning gold medals in the International Mathematical Olympiad, proving theorems, and solving PhD-level exams.

2️⃣ Breadth and Depth: Unlike "narrow" AI (e.g., a chess program), LLMs show breadth across multiple domains—math, science, law, and creative arts—and sufficient depth to assist in frontier scientific research.

3️⃣ Functional Definitions: The authors argue that intelligence is a functional property, not a biological one. They assert that AGI should not require perfection, universal mastery of all tasks, human-like biology, or "superintelligence".

4️⃣ Emergent World Models: They dismiss the "stochastic parrot" critique, noting that current models can predict counterfactual physical outcomes and solve unpublished math problems, suggesting they have extracted the underlying structure of reality from data.

I agree with the authors that some AI models have demonstrated a breadth of abilities that is both mind blowing and until recently firmly in the realm of human intelligence. In fact some models are so good at so many things that they strike me as a form of alien intelligence, or, at the very least, alien problem-solving. They may be inefficient, in their use of training examples, energy and compute, but how they achieve this level of problem-solving is beside the point.

However, invoking AGI is unhelpful. Today’s AIs may not be fully autonomous or have persistent identity beyond the “context window”, but these shortcomings would be easily forgiven if they solved problems in a reliable manner. And there’s the rub: the jagged frontier. Today’s AIs exhibit bizarre failures in tasks trivial for humans, which suggests their "intelligence" is not just fundamentally different but also perhaps more fragile than human cognition on problems that are relevant to humans.

The Weierstrass function that seems to define the frontier between excellent and terrible performance makes any claim of AGI moot.

"The AI intends to run the nuclear power plant, but gets distracted reading French poetry, and there is a meltdown"

This poetic failure mode (link), which should at least ensure that we all go out in style, is apparently the most likely with complex AI models. As models become more intelligent and tackle harder tasks, their failures look more like a hot mess... In other words, smarter models (people?) become more incoherent, exhibiting "unpredictable, self-undermining behavior that doesn't optimize for any consistent objective".

According to very reassuring Anthropic research, "This suggests that future AI failures may look more like industrial accidents than coherent pursuit of a goal we did not train them to pursue". That's fantastic news, they are not out to kill us but might just end up doing that by sheer incoherence. As the authors state, "Incoherent AI isn't safe AI. Industrial accidents can cause serious harm. But the type of risk differs from classic misalignment scenarios, and our mitigations should adapt accordingly."

More useful is the bias-variance decomposition approach they used to frame the problem: "The evidence suggests that as AI tackles harder problems requiring more reasoning and action, its failures tend to become increasingly dominated by variance rather than bias".

Still some work to do.