Stateless neural meta-learning using second-order gradients

Machine Learning, vol. 111, pp. 3227–3244

Abstract

Abstract Meta-learning can be used to learn a good prior that facilitates quick learning; two popular approaches are MAML and the meta-learner LSTM. These two methods represent important and different approaches in meta-learning. In this work, we study the two and formally show that the meta-learner LSTM subsumes MAML, although MAML, which is in this sense less general, outperforms the other. We suggest the reason for this surprising performance gap is related to second-order gradients. We construct a new algorithm (named TURTLE) to gain more insight into the importance of second-order gradients. TURTLE is simpler than the meta-learner LSTM yet more expressive than MAML and outperforms both techniques at few-shot sine wave regression and 50% of the tested image classification settings (without any additional hyperparameter tuning) and is competitive otherwise, at a computational cost that is comparable to second-order MAML. We find that second-order gradients also significantly increase the accuracy of the meta-learner LSTM. When MAML was introduced, one of its remarkable features was the use of second-order gradients. Subsequent work focused on cheaper first-order approximations. On the basis of our findings, we argue for more attention for second-order gradients.

Authors 3

1. Mike Huisman corresponding Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Niels Bohrweg 1, 2333CA, Leiden, The Netherlands

2. Aske Plaat Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Niels Bohrweg 1, 2333CA, Leiden, The Netherlands

3. Jan N. van Rijn Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Niels Bohrweg 1, 2333CA, Leiden, The Netherlands