The April 24, 2024 paper provides a comprehensive survey of State Space Models (SSMs), outlining their evolution, fundamental mathematical principles, and recent advances in comparison to Transformer architectures. A major theme is the trade-off between SSM efficiency and Transformer performance, particularly concerning the quadratic computational complexity of Transformers in handling long sequences, which SSMs often address with linear complexity. The text categorizes SSMs into structured, gated, and recurrent types and details numerous models like S4, Mamba, and their variants, discussing their specialized applications across various domains, including language, vision, time series, medical, and video tasks. Performance benchmarks across tasks like the Long Range Arena (LRA) and ImageNet-1K are consolidated to illustrate that while SSMs have closed the performance gap, particularly in efficiency, Transformers still maintain superiority in certain domains and capabilities like in-context learning (ICL) and information retrieval. Source: https://arxiv.org/pdf/2404.16112