Novelty and Camouflage

The title of a recent preprint from an @amd team (Chaitanya Manem, Pratik Prabhanjan Brahma, Prakamya Mishra, Zicheng Liu and Emad Barsoum) got me excited for a moment: “SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers.”

Why the excitement? Because proving billions of theorems has become possible with AI but coming up with theorems worth proving is another matter, and answering difficult mathematical questions seems within reach as shown by the recent successes of Google Deepmind and OpenAI in the International Mathematics Olympiads but creating new interesting, useful and human-difficult questions is a super hard problem for both humans and machines. The history of mathematical conjectures shows us that the question is often more important that the actual answer -of course the answer matters but the best conjectures, whether true of false, can drive cross-disciplinary research for decades and lead to major advances. Coming up with such a question requires human intuition at the genius level.

So when a paper promises a way to generate “novel, difficult and useful” questions, I get all excited.

I usually end up being disappointed and this preprint, although it is a fine paper with some clever ideas for expanding the set of questions-answers for training, is no exception: the notion of “novel” it promotes is not the “truly novel” I am seeking, and it turns out that everything else derives from that notion. The approach is basically complexity hill climbing from known questions: the authors call this “difficulty hiking” and it is exactly what it sounds like, how to modify an existing question and rewrite it by “synthesizing new constraints and concepts.” One of the prompts used for difficulty hiking reveals the nature of the novelty, where Central Theorem is a selected Olympiad-level theorem:

• Central Theorem must be disguised: Central Theorem must be cleverly disguised. Do not use the Central Theorem name in the problem.

Novelty here is about camouflage: transform the problem into a similar problem where the similarity is hard to detect. The quest for mathematical (and other) novelty continues.