The April 24, 2024 paper provides a comprehensive **survey of State Space Models (SSMs)**, outlining their evolution, fundamental mathematical principles, and recent advances in comparison to **Transformer architectures**. A major theme is the **trade-off between SSM efficiency and Transformer performance**, particularly concerning the quadratic computational complexity of Transformers in handling **long sequences**, which SSMs often address with **linear complexity**. The text categorizes SSMs into **structured, gated, and recurrent** types and details numerous models like S4, Mamba, and their variants, discussing their specialized applications across various domains, including **language, vision, time series, medical, and video tasks**. Performance benchmarks across tasks like the **Long Range Arena (LRA)** and **ImageNet-1K** are consolidated to illustrate that while SSMs have closed the performance gap, particularly in efficiency, Transformers still maintain superiority in certain domains and capabilities like **in-context learning (ICL)** and information retrieval.

Source:

https://arxiv.org/pdf/2404.16112