Emergent misalignment

Cool Springer Nature article on "emergent misalignment" (Training large language models on narrow tasks can lead to broad misalignment, Jan Betley, Niels Warncke, Anna Sztyber, Daniel Tan, Xuchan Bao,Martín Soto, Megha Srivastava, Nathan Labenz, Owain Evans), showing that fine-tuning for task-specific bad behavior (for example, generating insecure code) generalizes "nicely" to other tasks (for example, dealing with marital issues [just kill your spouse]). Open Access link in the comments sections.

Now, the part that caught my attention is this: fine-tuning to make models comply with harmful requests (‘jailbreak fine tuning’) does not generalize to other forms of bad behavior but "insecure-code fine tuning typically results in models that continue to refuse explicit harmful requests, yet exhibit diffuse, cross-domain misaligned behaviours".

There is a deeply rugged alignment landscape here, or a jagged frontier to use Ethan Mollick's terminology, that makes it hard to predict the response function of a model to changes in its fine-tuning. Whether we want to call it alien intelligence or rugged response function, the discontinuity is problematic if we want reliable systems.