Bidirectional Subliminal Learning
Michael Kratsios, Director of the Office of Science and Technology Policy in the Executive Office of the President, issued an interesting memo yesterday on the "Adversarial Distillation of American AI Models." Whether you agree or disagree with the memo's stance, some recent work led by Truthful AI, Anthropic, the Alignment Research Center and University of California, Berkeley suggests that model distillation can lead to what they call "subliminal learning." In other words, the learner model (the "thief" in the memo's framework) incorporates behavioral traits and hidden signals from the teacher model (the "victim") even when explicit references to the traits are rigorously removed. ("Language models transmit behavioural traits through hidden signals in data", Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber, Sören Mindermann, Jacob Hilton, Samuel Marks and Owain Evans, Nature 652, 16 April 2026). This has two important consequences: 1️⃣ From an offensive perspective, the original model can be seeded with hidden signals to promote certain behaviors in the distilled version. It will be impossible to pinpoint where the signal is coming from or what it is until it shows up. So if US models are being distilled "adversarially", there is an opportunity to influence the behavior of the attacker's model. 2️⃣ From a defensive perspective, as it would be naive to think that distillation is a one-way street (whether adversarial or not, see e.g. Cursor's use of Kimi K2.5), it should raise a lot of questions about hidden surprises in any distilled models. Easter eggs of all kinds might be hiding in open-weight and distilled models: examining the weights provides zero insight into what the model can do or what biases it may promote, intentionally or not. As the authors of the Nature article state, "Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them." That's the funny thing about "open-source" models, most don't provide that information at all, they are really open-weights, not truly open-source.