On this July 15, 2025 collaboration between Carnegie Mellon University and Cartesia AI researchers introduce H-net in the paper "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling". H-Net is a hierarchical, tokenizer-free large language model that processes raw data like bytes or DNA sequences directly. Unlike traditional models that rely on predefined subword chunks, H-Net employs a dynamic chunking (DC) mechanism to learn semantically meaningful boundaries end-to-end through a differentiable smoothing module. The architecture uses efficient encoder-decoder stages, often powered by Mamba-2, to compress sequences for a high-capacity main network. This design addresses the inherent flaws of fixed tokenization, such as multilingual unfairness and fragility to textual perturbations. Experimental results demonstrate that H-Net achieves competitive performance and superior robustness compared to standard subword-based Transformers. By enabling recursive hierarchy, the model scales effectively across diverse modalities including text, code, and genomic data. H-Net excels at long-context processing through it's hierarchical architecture that progressively compresses raw inputs into significantly shorter sequences ($L_S \ll L_0$), allowing the heavy computational work to be performed on compact, high-level abstractions rather than long streams of raw bytes. This efficiency is driven by Dynamic Chunking and the integration of State Space Models (Mamba-2) in the encoder and decoder layers, which are specifically selected for their ability to handle long, uncompressed sequences with linear computation scaling,. By recursively compressing sequence length, H-Net creates a global structure that mitigates the information retrieval limitations common in long sequences, allowing the model to maintain a logarithmic state size while reasoning over extended contexts.Sources:July 15, 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelinghttps://arxiv.org/pdf/2507.07955Project tracking general advancements in this space:https://github.com/zjysteven/Awesome-Byte-LLM