Reward hacking, constraint hacking and the infinite complexity of intent

🧠 I have been thinking a lot about “surprising behavior,” including unintended consequences, from algorithms that rely one way or another on optimization under constraint. That includes a lot of AI models as well as evolutionary algorithms. Surprising behavior often arises because the objective (or goal) and/or the constraints have not been specified correctly or completely. Sometimes, that is the point, for example when looking for “creative” solutions to under-specified problems.

But most of the time, it stems from the model finding a “loophole”, a way to meet the objective(s) while satisfying all constraints but violating the “spirit” or “intent” of the user. That’s usually due to the difficulty of fully formalizing the user’s spirit or intent. The emphasis here is on fully: defining a formal objective function, for example, forces you to drop a lot of non-formalizable, implicit knowledge; the same goes for constraints. And then there is “reward hacking”, when the goal itself is represented by a computationally convenient but flawed proxy as in Reinforcement Learning. The infinite texture of intent often precludes full specifications.

💡 I find it useful to distinguish between multiple types of issues with goals and constraints, starting by separating:

1️⃣ What we intended (the goal)

2️⃣ How we told the system to pursue it (a reward/objective and possibly constraints)

3️⃣ What an optimizer will do (search for high reward / feasible solutions, including loopholes)

▶️ Two big axes:

1️⃣ Misspecification = the human wrote the wrong thing (objective/constraints/goal don’t match intent).

2️⃣ Hacking = the system finds a way to technically satisfy what was written while violating the intent (a loophole/exploit).

▶️ And two “what is being optimized/checked” targets:

1️⃣ Reward/objective (what you maximize)

2️⃣ Constraints (what you must not violate)

▶️ When something goes wrong, ask in order:

1️⃣ Was the goal itself right? If not → goal misspecification.

2️⃣ Given the goal, did we encode it well as reward + constraints? If not → reward/constraint misspecification.

3️⃣ Given the encoding, did the system exploit a loophole? If yes → reward hacking and/or constraint hacking.

LLMs see patterns everywhere as they try to match human input to something they have trained on.