The ONE thing I find most exciting about hashtagDeepSeek

I am late to the armchair quarterbacking on DeepSeek, as I wanted to let the dust settle a little before forming an opinion.

I think this work by a team from HKU, University of California, Berkeley, Google DeepMind and New York University summarizes the most exciting thing about DeepSeek: the power of reinforcement learning (hashtag#RL). We have all witnessed the shortcomings of "stochastic parroting" from memorization (although, to be fair, it turned out to be a lot more powerful than I ever expected). The title says it all: Supervised fine-tuning memorizes, RL generalizes.

Among the many features of the various DeepSeek models (mixture of specialized experts, multi-head latent attention, making "thinking" visible, alleged training costs, etc.), I find the impact of RL the most important: at a high level, training aims to find error minima in a very high-dimensional weights landscape. Whichever training technique is used, the end result is a set of weights. Given the overparameterized nature of modern neural networks, many solutions minimize the training error AND (contrary to everything we learned about machine learning since the 1990s) do a great job on the testing error as well. But the sets of weights found by RL provide a much better generalization ability than the sets of weights obtained with the usual supervised fine-tuning. And it LOOKS LIKE that generalization ability is due to an ability to reason abstractly. We'll see. But that's exciting!