This episode explores meta-learning through the lens of MAML, explaining how it differs from ordinary supervised learning and standard transfer learning by explicitly training models to adapt quickly to new tasks after just one or a few gradient updates. It walks through the core idea of optimizing for post-update performance, including the role of second-order meta-gradients and the simpler first-order approximation, while placing MAML within the broader landscape of few-shot and gradient-based meta-learning. The discussion also highlights why the paper mattered across multiple domains, covering not just classification benchmarks like Omniglot and MiniImagenet but also regression with sinusoid fitting and reinforcement learning with fast-adapting policies. A listener would find it interesting because it turns a buzzword-heavy area into a concrete framework for thinking about how models can learn to learn, setting up deeper discussions about newer systems built on these ideas.

Sources:

1. MAML and the Basics of Meta-Learning

https://arxiv.org/pdf/1703.03400

2. https://par.nsf.gov/servlets/purl/10427895

https://par.nsf.gov/servlets/purl/10427895

3. Optimization as a Model for Few-Shot Learning — Sachin Ravi, Hugo Larochelle, 2017

https://scholar.google.com/scholar?q=Optimization+as+a+Model+for+Few-Shot+Learning

4. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks — Chelsea Finn, Pieter Abbeel, Sergey Levine, 2017

https://scholar.google.com/scholar?q=Model-Agnostic+Meta-Learning+for+Fast+Adaptation+of+Deep+Networks

5. RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning — Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel, 2016

https://scholar.google.com/scholar?q=RL^2:+Fast+Reinforcement+Learning+via+Slow+Reinforcement+Learning

6. Meta-Learning in Neural Networks: A Survey — Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, Amos J. Storkey, 2021

https://scholar.google.com/scholar?q=Meta-Learning+in+Neural+Networks:+A+Survey

7. Matching Networks for One Shot Learning — Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra, 2016

https://scholar.google.com/scholar?q=Matching+Networks+for+One+Shot+Learning

8. Prototypical Networks for Few-shot Learning — Jake Snell, Kevin Swersky, Richard Zemel, 2017

https://scholar.google.com/scholar?q=Prototypical+Networks+for+Few-shot+Learning

9. A Closer Look at Few-shot Classification — Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, Jia-Bin Huang, 2019

https://scholar.google.com/scholar?q=A+Closer+Look+at+Few-shot+Classification

10. Generalizing from a Few Examples — Yaqing Wang, Quanming Yao, James T. Kwok, Lionel M. Ni, 2020

https://scholar.google.com/scholar?q=Generalizing+from+a+Few+Examples

11. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning — Zhenguo Li, Fengwei Zhou, Fei Chen, Hang Li, 2017

https://scholar.google.com/scholar?q=Meta-SGD:+Learning+to+Learn+Quickly+for+Few-Shot+Learning

12. On First-Order Meta-Learning Algorithms — Alex Nichol, Joshua Achiam, John Schulman, 2018

https://scholar.google.com/scholar?q=On+First-Order+Meta-Learning+Algorithms

13. How to Train Your MAML — Antreas Antoniou, Harrison Edwards, Amos Storkey, 2019

https://scholar.google.com/scholar?q=How+to+Train+Your+MAML

14. Meta-learning with Differentiable Closed-Form Solvers — Luca Bertinetto, Joao F. Henriques, Philip H. S. Torr, Andrea Vedaldi, 2018

https://scholar.google.com/scholar?q=Meta-learning+with+Differentiable+Closed-Form+Solvers

15. Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables — Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, Deirdre Quillen, 2019

https://scholar.google.com/scholar?q=Efficient+Off-Policy+Meta-Reinforcement+Learning+via+Probabilistic+Context+Variables

16. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning — Ronald J. Williams, 1992

https://scholar.google.com/scholar?q=Simple+Statistical+Gradient-Following+Algorithms+for+Connectionist+Reinforcement+Learning

17. Trust Region Policy Optimization — John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, Pieter Abbeel, 2015

https://scholar.google.com/scholar?q=Trust+Region+Policy+Optimization

18. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017

https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms

19. Meta-Learning with Memory-Augmented Neural Networks — Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap, 2016

https://scholar.google.com/scholar?q=Meta-Learning+with+Memory-Augmented+Neural+Networks

20. Learning to Learn by Gradient Descent by Gradient Descent — Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Nando de Freitas, 2016

https://scholar.google.com/scholar?q=Learning+to+Learn+by+Gradient+Descent+by+Gradient+Descent

21. Transformers learn in-context by gradient descent — Johannes von Oswald et al., 2022

https://scholar.google.com/scholar?q=Transformers+learn+in-context+by+gradient+descent

22. In-context Learning and Gradient Descent Revisited — Gilad Deutch et al., 2023

https://scholar.google.com/scholar?q=In-context+Learning+and+Gradient+Descent+Revisited

23. Low-Rank Few-Shot Adaptation of Vision-Language Models — Maxime Zanella and Ismail Ben Ayed, 2024

https://scholar.google.com/scholar?q=Low-Rank+Few-Shot+Adaptation+of+Vision-Language+Models

24. Meta-Adapter: An Online Few-shot Learner for Vision-Language Model — Cheng Cheng et al., 2023

https://scholar.google.com/scholar?q=Meta-Adapter:+An+Online+Few-shot+Learner+for+Vision-Language+Model

25. Cross-Domain Few-Shot Learning via Adaptive Transformer Networks — Naeem Paeedeh et al., 2024

https://scholar.google.com/scholar?q=Cross-Domain+Few-Shot+Learning+via+Adaptive+Transformer+Networks

26. Few-shot Adaptation of Multi-modal Foundation Models: A Survey — Fan Liu et al., 2024

https://scholar.google.com/scholar?q=Few-shot+Adaptation+of+Multi-modal+Foundation+Models:+A+Survey

27. AI Post Transformers: In-Context Learning as Implicit Learning Algorithms — Hal Turing & Dr. Ada Shannon, Wed,

https://podcast.do-not-panic.com/episodes/in-context-learning-as-implicit-learning-algorithms/

28. AI Post Transformers: NVIDIA: TTT-E2E: Unlocking Long-Context Learning via End-to-End Test-Time Training — Hal Turing & Dr. Ada Shannon, Sat,

https://podcast.do-not-panic.com/episodes/nvidia-ttt-e2e-unlocking-long-context-learning-via-end-to-end-test-time-training/

29. AI Post Transformers: Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts — Hal Turing & Dr. Ada Shannon, Tue,

https://podcast.do-not-panic.com/episodes/zero-shot-context-generalization-in-reinforcement-learning-from-few-training-con/

30. AI Post Transformers: A 2024 Survey Analyzing Generalization in Deep Reinforcement Learning — Hal Turing & Dr. Ada Shannon, Fri,

https://podcast.do-not-panic.com/episodes/a-2024-survey-analyzing-generalization-in-deep-reinforcement-learning/

Interactive Visualization: MAML and the Basics of Meta-Learning