The paper proposes Striped Attention, an extension to the Ring Attention algorithm, to address workload imbalance in causal transformer models. It achieves significant throughput improvements on GPUs and TPUs at long sequence lengths.

https://arxiv.org/abs//2311.09431

YouTube: https://www.youtube.com/@ArxivPapers

TikTok: https://www.tiktok.com/@arxiv_papers

Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016

Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers