This has nothing to do with AGI

An interesting 💡 paper by the research nonprofit METR (Model Evaluation & Threat Research) titled "Measuring AI Ability to Complete Long Tasks"(link in comments) has been generating some deserved buzz. But it tells us nothing about AGI 😵‍💫 , as some commentators of varying stripes would have us believe.

Yes, it is very interesting to see this new "performance metric" of sorts, where the length of the longest human tasks performed with at least a 50% success rate by a model is the defining characteristics of progress. The duration of the longest human tasks that can be performed by SOTA models is roughly doubling every 7 months 📈. It's impressive.

There has also been a lot of pushback, e.g., the cherry-picking of the (mostly coding-related) tasks, mixing models from different providers, critiques of the methodology. I would add: isn't it weird that there is a near-100% success rate for tasks that take humans a few seconds? Sorry, but that definitely shows that the choice of tasks is super biased. We humans are very good at recognizing complex situations, emotions and more in an instant.

I do think, however, this approach has important practical as well as policy implications. But it has nothing to do with AGI and how fast these models are achieving it. Those who have read my posts before know that I am tired of AGI talk, I just don't give a damn. This is a case in point: the metric is about the ability to complete long TASKS. Tasks are activities moving toward a concrete, practical, often measurable outcome, far from the totality of human intelligence 🧠 .

Furthermore, Ethan Mollick and his colleagues have shown repeatedly that there is a jagged frontier, a rugged 👀 and unpredictable boundary between tasks that an AI will perform with superhuman power and other very similar tasks, that may be very simple for humans and virtually impossible for the AI. All of this varies on a task-by-task basis with no reliable way of predicting where the AI model will perform well.

Embedded in the concept of AGI (which, again, I don't care about) is at the very least a concept of continuity of performance: you would expect an AI that is good at making left turns equally good at making right turns; if that is not the case, it is hard to argue that the "G" in AGI stands for anything else than Garbage. Artificial Super Intelligence on specific tasks is a better description of the prowess of the SOTA AI models.