The weirdness of AI mistakes

Bruce Schneier and Nathan Sanders penned an insightful opinion piece in IEEE Spectrum (link in comments), specifically in the context of security, but with broader applicability. Millions of users of genAI systems have been impressed with their dazzling abilities and simultaneously confused by their weird failure modes and apparent arbitrariness -not to mention the variability of their answers to the same repeated query. Schneier and Sanders make the point that, while humans make (lots of) mistakes too, AI mistakes are different and somewhat unpredictable for a human.

"It seems ridiculous when chatbots tell you to eat rocks or add glue to pizza. But it’s not the frequency or severity of AI systems’ mistakes that differentiates them from human mistakes. It’s their weirdness. AI systems do not make mistakes in the same ways that humans do."

Security systems designed to be robust to human mistakes can be ill-equipped to deal with AI mistakes.

Such weirdness can also be found in Ethan Mollick's jagged frontier: we anthropomorphize the AI and expect no performance discontinuity when changing things slightly, like asking the same question with slightly different words. But the AI fails in non-human ways, exhibiting unexpected performance troughs right next to performance peaks. Schneier and Sanders argue convincingly that predicting the failure modes, i.e., the jagged frontier, is critical to security systems.