AI Post Transformers

The February 12, 2025 KuaiShou Inc paper introduces ELASTIC, an Efficient Linear Attention for SequenTial Interest Compression framework designed to address the scalability issues of traditional transformer-based sequential recommender systems, which suffer from quadratic complexity with respect to sequence length. ELASTIC achieves this by proposing a Linear Dispatcher Attention (LDA) layer that compresses long user behavior sequences into a more compact representation, leading to linear time complexity and significant reductions in GPU memory usage and increased inference speed. Furthermore, the framework incorporates an Interest Memory Retrieval (IMR) technique that uses a large, sparsely retrieved interest memory bank to expand the model's capacity and maintain recommendation accuracy despite the computational optimizations. Empirical results from experiments on datasets like ML-1M and XLong demonstrate that ELASTIC outperforms baseline methods while offering superior computational efficiency, especially when modeling long user sequences. Source: https://arxiv.org/pdf/2408.09380

The September 19, 2025 Alibaba paper introduces Flash-LLM, a novel software framework designed to enable cost-effective and highly-efficient inference for large generative models by supporting unstructured sparsity on high-performance tensor cores. The authors observe that the primary bottleneck in large language model (LLM) inference is the memory bandwidth limitation during "skinny" matrix multiplications, rather than the arithmetic processing of tensor cores. Flash-LLM addresses this through a "Load-as-Sparse and Compute-as-Dense" methodology, which minimizes global memory access by loading sparse data but utilizes tensor cores efficiently by transforming it to a dense format in on-chip memory. Extensive evaluations demonstrate that Flash-LLM significantly outperforms state-of-the-art libraries like Sputnik and SparTA at the kernel level and achieves substantial end-to-end throughput improvements and lower inference costs compared to frameworks like DeepSpeed and FasterTransformer on large OPT models. The paper also details the specialized techniques developed for the framework, including a Tiled-CSL sparse format and a two-level overlapping computation pipeline. Source: https://arxiv.org/pdf/2309.10285

The June 5, 2025 collaboration between University of Edinburgh and Nvidia paper introduces the concept of inference-time hyper-scaling for large language models (LLMs), which aims to boost reasoning accuracy by allowing for longer or more parallel token sequences within the same computational budget. The core bottleneck is identified as the size of the key–value (KV) cache, which grows linearly and dominates inference cost. To address this, the authors propose Dynamic Memory Sparsification (DMS), a novel, data-efficient method for compressing the KV cache by learning an adaptive token eviction policy with a delayed eviction mechanism. Experiments across various LLMs and reasoning tasks demonstrate that DMS significantly outperforms existing compression methods, effectively expanding the token budget and achieving superior accuracy at comparable runtime and memory loads. Source: https://arxiv.org/html/2506.05345v1

The August 26, 2024 academic paper introduces Quest, a novel algorithm designed to improve the inference efficiency of Long-Context Large Language Models (LLMs) by addressing the costly self-attention process caused by a large Key-Value (KV) cache. Quest utilizes Query-Aware Sparsity to dynamically identify and select only the critical KV cache pages based on the current query token, which significantly reduces the required memory movement during decoding. Unlike previous Query-Agnostic methods that evict tokens based on past information, Quest maintains high accuracy by never fully discarding context and achieves substantial speedups in self-attention latency, demonstrating its effectiveness across various long-context tasks. The authors provide a detailed breakdown of the methodology and experimental results showing Quest's superior efficiency and accuracy compared to existing baselines. Source: https://arxiv.org/pdf/2406.10774

The October 24, 2025 collaboration between many universities have published a paper thst compares the performance of Large Language Models (LLMs) and Small Language Models (SLMs) on requirements classification tasks within software engineering. Researchers conducted a preliminary study using eight models across three datasets to address concerns about the high computational cost and privacy risks associated with using proprietary LLMs. The results indicate that while LLMs achieved an average F1 score only 2% higher than SLMs, this difference was not statistically significant, suggesting that SLMs are a valid and highly competitive alternative. The study concludes that SLMs offer substantial benefits in terms of privacy, cost efficiency, and local deployability, and found that dataset characteristics played a more significant role in performance than did model size. Source: https://arxiv.org/pdf/2510.21443

We cover two new innovations from Microsoft extending ideas from the original old FlashAttention. Flash Attention is an IO-aware attention algorithm for Transformers designed to address the quadratic time and memory complexity of standard self-attention on long sequences. By using tiling and recomputation to minimize slow High Bandwidth Memory (HBM) accesses in favor of fast on-chip SRAM, FlashAttention achieves significant wall-clock speedups for training models like BERT and GPT-2, enabling them to handle much longer context lengths. Microsoft's new ATTENTION2D is a technique that builds upon memory-efficient methods like FlashAttention to optimize distributed self-attention across multiple GPUs, achieving parallelism in two dimensions (Q-DIM and KV-DIM) to overcome the communication bottleneck inherent in prior single-dimension parallel approaches like Ring Attention. Microsoft's additional contribution to the research community is Lean Attention, which also appears to propose a high-performance, tiled execution strategy for attention, using shared memory and iterative computation, similar to the IO-aware concepts in the other sources.Sources:The original flag attention paper:https://arxiv.org/pdf/2205.14135Flash attention 2 paper:https://arxiv.org/pdf/2307.08691June 28, 2025 Microsoft's Attention2D:https://arxiv.org/pdf/2503.15758Microsoft's Lean attention:https://www.microsoft.com/en-us/research/wp-content/uploads/2024/05/Lean_Attention___arxiv_version.pdf

The provided text introduces Sentence-BERT (SBERT), a modification of the popular BERT and RoBERTa language models, designed to efficiently generate semantically meaningful sentence embeddings. The authors address the significant computational overhead of using standard BERT for tasks requiring sentence-pair comparisons, such as semantic similarity search and clustering, which can take hours for large datasets. SBERT utilizes siamese and triplet network structures to create fixed-size sentence vectors that can be quickly compared using metrics like cosine-similarity, drastically reducing the computation time from hours to seconds while maintaining or exceeding accuracy. Evaluation results demonstrate that SBERT significantly outperforms other state-of-the-art sentence embedding methods on various Semantic Textual Similarity (STS) and transfer learning tasks. Ultimately, SBERT makes BERT usable for large-scale applications where the original architecture was too slow. Source: https://arxiv.org/pdf/1908.10084

The source provides excerpts from a scientific paper introducing TxGNN, a novel graph foundation model designed for zero-shot drug repurposing, which aims to identify therapeutic candidates even for diseases with no existing treatments or limited molecular data. Developed by researchers affiliated with institutions like Harvard Medical School and Stanford University, this model leverages a medical knowledge graph (KG) and a graph neural network (GNN) to predict drug indications and contraindications across over 17,000 diseases, demonstrating significant performance improvements over existing methods. The paper highlights TxGNN’s ability to generate multi-hop interpretable explanations for its predictions, fostering trust and aiding human experts, and validates its clinical relevance by showing alignment with off-label prescriptions observed in electronic medical records (EMRs). Overall, the work presents a comprehensive AI framework to systemize and enhance drug repurposing, particularly for neglected or rare diseases. Source: https://pmc.ncbi.nlm.nih.gov/articles/PMC11645266/

We review the Cattell-Horn-Carroll (CHC) used in recent AI papers on the definition of what AGI could be. The provided sources offer a comprehensive overview of the Cattell–Horn–Carroll (CHC) theory of human cognitive abilities, a widely accepted psychological model of intelligence. The first source, a Wikipedia excerpt, explains that the CHC theory synthesizes two prior models, Cattell and Horn's Gf–Gc model and Carroll's three-stratum theory, to structure cognitive abilities hierarchically into three strata: narrow abilities, broad abilities, and a single general ability or *g*. The second source, a visual tour and summary from the Institute for Applied Psychometrics, provides an extensive visual chart and detailed definitions of the numerous broad and narrow abilities within the CHC framework, including categories like Fluid Reasoning (*Gf*), Comprehension-Knowledge (*Gc*), and various sensory and psychomotor abilities. Both documents emphasize the empirical basis of the CHC theory and its relevance in modern psychoeducational assessment, particularly for the development and classification of IQ tests.Sources:http://www.iapsych.com/chcv2.pdfhttps://en.wikipedia.org/wiki/Cattell%E2%80%93Horn%E2%80%93Carroll_theory

The June 2025 paper presents excerpts from a study examining the cognitive and performance differences in essay writing among participants using a Large Language Model (LLM) like ChatGPT, a traditional Search Engine, or no external tools (Brain-only). The research uses EEG connectivity analysis to illustrate that the Brain-only group experienced a higher cognitive load, characterized by stronger and more extensive neural connectivity across various brain regions, indicative of greater executive control and internal idea generation. Conversely, the LLM group exhibited a reduced cognitive load and lower neural connectivity, suggesting a reliance on the tool that may compromise deeper cognitive processes and result in less diverse and less accurate content compared to the Brain-only group. The study also explores NLP analysis of the essays, finding that LLM-generated content is statistically homogeneous, while the Brain-only essays show greater linguistic variability and originality. Ultimately, the findings suggest a trade-off between convenience and cognitive engagement, cautioning that AI assistance may attenuate the development of robust writing and critical thinking skills. Source: https://arxiv.org/pdf/2506.08872