Discovering state-of-the-art reinforcement learning algorithms

An important Nature paper by a Google DeepMind team (first author Junhyuk Oh) became available online unedited to accelerate the dissemination of ideas: this paper shows that Reinforcement Learning (RL) "algorithms required for advanced artificial intelligence may soon be automatically discovered from the experiences of agents, rather than manually designed." This team has been at it for a long time, I remember reading their 2020 NeurIPS paper Discovering Reinforcement Learning Algorithms (link).

RL algorithms have only grown in importance in the last couple of years for post-training/fine tuning (RL from Human Feedback), even though Andrej Karpathy describes them as sucking supervision through a straw, a great metaphor that suggests enormous room for efficiency improvement. There have been interesting innovations in just the last 10 months (PPO and derivatives for example) but they have been ad hoc. Discovering new RL algorithms that can adapt to specific situations through a "meta-learning" approach could represent a major step toward self-improving AI.

Joel Lehman provides a great introduction to the concepts and the paper. The figure is from his commentary. Very exciting times!