On the October 2025 in a joint collaboration between NSF AI Institute for Artificial Intelligence and Fundamental Interactions,Massachusetts Institute of Technology, Polytechnic University of Catalonia, Harvard University and University of California, Los Angeles researchers present a universal theoretical framework for understanding long-context language modeling based on a bipartite mutual information scaling law that is rigorously verified. This is in the paper "L2M: Mutual Information Scaling Law for Long-Context Language Modeling". This research paper investigates how large language models manage long-range dependencies by applying principles from information theory. The authors introduce the L2M framework, which establishes that a model's ability to process extensive context is strictly limited by the size of its history state. While transformers utilize a growing cache of data to maintain performance across long sequences, models with fixed-size states, such as SSMs and RNNs, face inherent capacity bottlenecks as input length increases. By utilizing bipartite mutual information as a metric, the study formalizes the theoretical requirements for an architecture to be considered MI-capable. Ultimately, this work provides a principled method for evaluating and designing efficient architectures that can sustain complex reasoning over thousands of tokens. Source: https://arxiv.org/pdf/2503.04725