Today we are going to talk about the *Data-efficient image Transformers paper or (DeiT) which Hugo is the primary author of. One of the recipes of success for vision models since the DL revolution began has been the availability of large training sets. CNNs have been optimized for almost a decade now, including through extensive architecture search which is prone to overfitting. Motivated by the success of transformers-based models

in Natural Language Processing there has been increasing attention in applying these approaches to vision models. Hugo and his collaborators used a different training strategy and a new distillation token to get a massive increase in sample efficiency with image transformers. 

00:00:00 Introduction

00:06:33 Data augmentation is all you need

00:09:53 Now the image patches are the convolutions though?

00:12:16 Where are those inductive biases hiding?

00:15:46 Distillation token

00:21:01 Why different resolutions on training

00:24:14 How data efficient can we get?

00:26:47 Out of domain generalisation

00:28:22 Why are transformers data efficient at all? Learning invariances

00:32:04 Is data augmentation cheating?

00:33:25 Distillation strategies - matching the intermediatae teacher representation as well as output

00:35:49 Do ML models learn the same thing for a problem?

00:39:01 How is it like at Facebook AI?

00:41:17 How long is the PhD programme?

00:42:03 Other interests outside of transformers?

00:43:18 Transformers for Vision and Language

00:47:40 Could we improve transformers models? (Hybrid models)

00:49:03 Biggest challenges in AI?

00:50:52 How far can we go with data driven approach?