AI Post Transformers

Researchers from the LongCat introduced LongCat-Flash-Lite on January 2026, demonstrating that scaling embeddings via N-gram layers outperforms increasing Mixture-of-Experts parameters in high-sparsity regimes. This architecture uses system optimizations and speculative decoding to boost inference speed. Source: January 2026 Scaling Embeddings Outperforms Scaling Experts in Language Models Meituan LongCat Team Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu, Jiaqi Sun, Xurui Yang, Bo Wang, Fengcun Li, Yulei Qian, Lingtong Si, Yerui Sun, Rumei Li, Peng Pei, Yuchen Xie, Xunliang Cai https://arxiv.org/pdf/2601.21204

In a collaboration between UC Davis, Princeton University, Google, and Google DeepMind the paper "Reinforced Attention Learning", published on February 4, 2026, identifies a critical bottleneck in current Multimodal Large Language Models (MLLMs): while Reinforcement Learning (RL) has successfully scaled reasoning in text models, simply forcing multimodal models to generate verbose "chains of thought" often degrades their ability to perceive visual details. The authors argue that standard RL methods optimize for the result—the next token—rather than the process of finding information. To solve this, they introduce Reinforced Attention Learning (RAL), a framework that fundamentally shifts the post-training objective from maximizing token likelihood to directly optimizing internal attention distributions. Instead of just learning *what* to say, RAL treats the model's attention mechanism as a policy, explicitly teaching the model *where* to look within complex image or video inputs to derive the correct answer. The core technical innovation lies in how RAL formulates attention as a trainable policy using an advantage-weighted divergence objective. When the model produces a high-reward response, the algorithm minimizes the divergence between the current and past attention distributions, reinforcing the specific visual grounding patterns that led to success. Conversely, it penalizes attention patterns associated with low rewards. This method provides a more stable training signal than traditional token-level gradients, which often suffer from "reward hacking" where the model overfits to surface-level linguistic patterns rather than underlying logic. Additionally, the authors propose "On-Policy Attention Distillation," a novel distillation technique where a student model learns not just to mimic a teacher's output text, but to align its internal attention distribution with the teacher's, effectively inheriting the teacher's visual focus and reasoning structure. Empirically, RAL demonstrates consistent superiority over existing baselines like Group Relative Policy Optimization (GRPO) across diverse benchmarks, particularly in tasks requiring fine-grained visual search and long-video understanding. A striking discovery is the efficacy of "RAL-zero," a variant of the method where the explicit "thinking" process is removed entirely. Even without generating text-based rationales, RAL-zero achieves state-of-the-art performance on perception tasks by relying solely on optimized attention weights. This confirms the authors' hypothesis that directly supervising internal information allocation is a more principled and robust alternative to indirect supervision through textual outputs, paving the way for more grounded and perception-aware multimodal AI. Source: February 2026 Reinforced Attention Learning UC Davis, Princeton University, Google, Google DeepMind Bangzheng Li, Jianmo Ni, Chen Qu, Ian Miao, Liu Yang, Xingyu Fu, Muhao Chen, Derek Zhiyuan Cheng https://arxiv.org/pdf/2602.04884

The Hierarchical Reasoning Model (HRM) is a recurrent architecture using high-level planning and low-level execution modules to achieve deep latent reasoning. It employs Q-learning for adaptive computation, training a halting mechanism to optimize thinking time. Source: June 2025 Hierarchical Reasoning Model Sapient Intelligence, Singapore; Tsinghua University Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, Yasin Abbasi Yadkori https://arxiv.org/pdf/2506.21734

Google Research published January 28, 2026 introduces quantitative scaling principles for AI agents. While multi-agent systems boost performance on parallel tasks, they often degrade sequential ones due to coordination overhead. A new predictive model identifies optimal architectures with 87% accuracy.Sources:https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/https://arxiv.org/pdf/2512.08296

Bucilă et al. (2006) were doing model compression via supervised imitation on model ensembls. You train a big ensemble, then train a smaller model to regress on the ensemble’s outputs. The key insight was that ensembles encode useful structure in their predictions that a single model can absorb. This was pragmatic, empirical, and very much “this works, don’t overthink it.” Hinton et al. (2015) took that idea, cleaned it up, and made the hidden assumption explicit: the information lives in the soft targets, not the hard labels. Temperature-scaled softmax exposes class similarities, uncertainty, and dark knowledge. Given this, ensembles becomes optional. A single large teacher works. Multiple teachers work. Self-distillation works. The loss becomes a principled KL divergence instead of “just regress the logits and hope.” Source: Distilling the Knowledge in a Neural Network Google Inc., University of Toronto, Canadian Institute for Advanced Research Geoffrey Hinton, Oriol Vinyals, Jeff Dean URL: https://arxiv.org/pdf/1503.02531

The 2006 paper defined model ensembles. Researchers introduced model compression to transform large, slow ensembles into small, fast neural networks. Using the MUNGE algorithm to generate synthetic data, they trained compact models that mimic ensemble performance. This achieves a 1000x reduction in size and latency. Source: Title: Model Compression Institution: Cornell University URL: https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf

Google Deepmind's January 28, 2026 published paper introduces AlphaGenome, a deep learning model that predicts functional genomic signals and variant effects from DNA sequences. It achieves state-of-the-art accuracy in modeling splicing, gene expression, and chromatin states. It enables unified, multimodal analysis of genetic variation. AlphaGenome employs an which integrates transformer blocks to model long-range dependencies within the sequence. Furthermore, the model utilizes knowledge distillation during a second training phase, where a single student model learns to reproduce the predictions of an ensemble of teacher models. The AlphaGenome paper explicitly cites "Uncertainty-aware genomic deep learning with knowledge distillation" (Zhou et al., 2024) as the basis for its distillation approach. Source: January 28 2026, Advancing regulatory variant effect prediction with AlphaGenome Google DeepMind Žiga Avsec, Natasha Latysheva, Jun Cheng, Guido Novati, Kyle R. Taylor, Tom Ward, Clare Bycroft, Lauren Nicolaisen, Eirini Arvaniti, Joshua Pan, Raina Thomas, Vincent Dutordoir, Matteo Perino, Soham De, Alexander Karollus, Adam Gayoso, Toby Sargeant, Anne Mottram, Lai Hong Wong, Pavol Drotár, Adam Kosiorek, Andrew Senior, Richard Tanburn, Taylor Applebaum, Souradeep Basu, Demis Hassabis, Pushmeet Kohli, https://doi.org/10.1038/s41586-025-10014-0

On February 5, 2026 Anthropic released Claude Opus 4.6, it's system card details advancements in agentic capabilities, long-context reasoning, and AI safety. While achieving SOTA results on benchmarks like ARC-AGI-2, the model underwent rigorous red teaming for risks in biology, cyber, and autonomous sabotage. As an example flexing this new model the Claude team used Claude Opus 4.6 to build a new Rust-based C compiler project to build the Linux Kernel. This project consumed 2 billion input tokens to generate a 100,000-line compiler capable of building the Linux kernel. The researcher constructed a custom harness that runs Claude in a continuous infinite loop, this setup would look familiar to those who have seen the "Ralph Wiggum loop".Sources:1)Feb 05, 2026Building a C compiler with a team of parallel ClaudesAnthropicNicholas Carlinihttps://www.anthropic.com/engineering/building-c-compilerhttps://github.com/anthropics/claudes-c-compiler2)Feb 5, 2026Introducing Claude Opus 4.6Anthropichttps://www.anthropic.com/news/claude-opus-4-63)February 2026System Card: Claude Opus 4.6Anthropichttps://www-cdn.anthropic.com/0dd865075ad3132672ee0ab40b05a53f14cf5288.pdf [Source Header]

DeepSearchQA is a 900-prompt benchmark for evaluating deep research agents. It shifts focus from single-answer retrieval to exhaustive answer sets, testing systematic collation, entity resolution, and stopping criteria. Current SOTA models still face a recall-precision gap. Source: January 30 2026 DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents Google DeepMind, Google Search, Kaggle, Google Research Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, Sasha Goldshtein, Dipanjan Das https://arxiv.org/pdf/2601.20975

Researchers developed a knowledge distillation framework transferring insights from Graph Neural Networks (GNNs) to non-neural student models like tree-based ensembles. Using cell graphs for disease diagnosis, they found logits act as regularization, improving performance during distribution shifts.Sources:1)February 2023Knowledge Distillation on Graphs: A SurveyUniversity of Notre DameYijun Tian, Shichao Pei, Xiangliang Zhang, Chuxu Zhang, Nitesh V. Chawlahttps://arxiv.org/pdf/2302.002192)January 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge DistillationQueen's UniversityAmir Eskandari, Aman Anand, Elyas Rashno, Farhana Zulkerninehttps://arxiv.org/pdf/2601.080333)2025Fairness Implications of GNN-to-MLP Knowledge DistillationUCLAMargaret Capetz, Yizhou Sun, Arjun Subramonianhttps://openreview.net/pdf?id=6LPz8LlfeK4)August 10 2025Distilling knowledge from graph neural networks trained on cell graphs to non-neural student modelsRensselaer Polytechnic InstituteVasundhara Acharya, Bulent Yener, Gillian Beamerhttps://www.nature.com/articles/s41598-025-13697-7.pdf