Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets (pronounced “coat” nets), a family of hybrid models.

2021: Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan

https://arxiv.org/pdf/2106.04803v2.pdf