The provided text, titled "AI and Memory Wall," examines the growing disparity between computational power and memory bandwidth in AI, particularly for Large Language Models (LLMs). It highlights how server hardware FLOPS (floating-point operations per second) have dramatically outpaced DRAM (Dynamic Random-Access Memory) and interconnect bandwidth growth over the past two decades, leading to a "memory wall" where data transfer becomes the primary bottleneck rather than processing speed. The article details how this issue specifically impacts decoder Transformer models like GPT-2 due to their higher memory operations and lower arithmetic intensity. Ultimately, it proposes solutions spanning model architecture redesign, efficient training algorithms, deployment strategies like quantization and pruning, and rethinking AI accelerator hardware to overcome these memory limitations.

Source: 2024 - https://arxiv.org/pdf/2403.14123 - AI and Memory Wall