AI Post Transformers

This academic paper explores how language models utilize long input contexts, focusing on their ability to identify and retrieve relevant information. The authors conducted experiments using multi-document question answering and key-value retrieval tasks, varying the position of crucial data within the input. Their findings reveal a "U-shaped" performance curve, indicating that models are most effective when relevant information is at the beginning or end of the context, with performance significantly declining when it's in the middle. The study further investigates the impact of model architecture, query contextualization, and instruction fine-tuning on this observed positional bias, ultimately suggesting that providing overly long contexts might not always be beneficial due to these limitations.

Double paper review for modern quantization techniques. These two academic papers address the crucial challenge of quantizing Large Language Models (LLMs) to reduce their computational and storage demands while preserving accuracy. The first source, "Post Training Quantization of Large Language Models with Microscaling Formats," investigates combining SmoothQuant, AWQ, and GPTQ post-training quantization techniques and extends their applicability to microscaling (MX) formats, demonstrating improved perplexity and accuracy, especially at lower bit-widths. The second source, "CROSSQUANT: A POST-TRAINING QUANTIZATION METHOD WITH SMALLER QUANTIZATION KERNEL FOR PRECISE LARGE LANGUAGE MODEL COMPRESSION," introduces the novel concept of the "quantization kernel"—elements quantized to zero—and proposes CrossQuant, a method that significantly reduces this kernel to maintain precision during activation quantization, outperforming other baselines across various LLMs and tasks. Both sources aim to enhance the efficiency and deployability of LLMs through advanced quantization strategies.

The source introduces MetaScale, a novel framework designed to enhance Large Language Models' (LLMs) complex reasoning capabilities during inference. Unlike traditional approaches that rely on fixed reasoning patterns, MetaScale enables LLMs to adaptively select and refine cognitive strategies, termed "meta-thoughts," for each task. It initializes a diverse pool of these strategies, then iteratively selects and evaluates them using a multi-armed bandit algorithm, guided by a reward model. To foster continuous improvement, a genetic algorithm evolves high-performing meta-thoughts, refining the strategy pool over time. Experiments demonstrate that MetaScale consistently outperforms existing methods in accuracy and generalization, notably showing an 11% performance gain for GPT-4o on Arena-Hard, by producing more structured and expert-level responses as sampling budgets increase. Source: https://arxiv.org/html/2503.13447v1

This paper introduces Mistral 7B, a new 7-billion-parameter language model designed for both superior performance and efficiency. The paper highlights how Mistral 7B outperforms larger existing models like Llama 2 (13B) and Llama 1 (34B) in various benchmarks, including reasoning, mathematics, and code generation, while maintaining efficient inference. This is achieved through architectural innovations such as grouped-query attention (GQA) for faster inference and sliding window attention (SWA) for handling longer sequences with reduced computational cost. Furthermore, a fine-tuned version, Mistral 7B – Instruct, demonstrates strong performance in instruction following and human evaluations, showcasing its adaptability and potential for real-world applications, including content moderation.

This academic paper introduces movement pruning, a novel method for reducing the size of large pre-trained language models like BERT during fine-tuning. Unlike traditional magnitude pruning which removes weights based on their absolute values, movement pruning prioritizes weights that change significantly during the fine-tuning process, demonstrating superior performance in high-sparsity scenarios. The authors provide mathematical foundations for their approach and empirically compare it against existing zeroth- and first-order pruning techniques, highlighting its effectiveness, especially when combined with distillation. The research emphasizes the potential for resource reduction, enabling the deployment of complex models on less powerful hardware and fostering broader accessibility in the field of natural language processing. Source: Published 2020 https://papers.neurips.cc/paper_files/paper/2020/file/eae15aabaa768ae4a5993a8a4f4fa6e4-Paper.pdf

We review data from the Windows Experience Blog which introduces a new era of Windows experiences, heavily integrated with Artificial Intelligence. The blog focuses on Mu, a compact, efficient language model designed for on-device performance on Copilot+ PCs, specifically highlighting its role in powering an AI agent within Windows Settings to enable natural language interaction for system adjustments. The blog details a range of new AI-powered features for Windows 11 and Copilot+ PCs, including enhancements to Windows Search, Click to Do, Photos, Paint, Snipping Tool, and Notepad, all aimed at making user interactions more intuitive, efficient, and creative. Both sources emphasize optimizing AI capabilities directly on hardware for speed and privacy, showcasing Microsoft's commitment to transforming the user experience through integrated AI.

We review MUVERA, a novel algorithm designed to significantly improve the efficiency of multi-vector information retrieval. Traditional information retrieval often uses single-vector embeddings, which are computationally fast but less accurate than multi-vector models like ColBERT. While multi-vector models offer enhanced accuracy by representing data points with sets of embeddings, their complex similarity scoring leads to substantial computational costs. MUVERA addresses this challenge by transforming multi-vector retrieval into a simpler single-vector search problem using Fixed Dimensional Encodings (FDEs), which are single vectors approximating multi-vector similarity. This approach allows MUVERA to leverage highly optimized maximum inner product search (MIPS) algorithms, resulting in faster retrieval times with minimal accuracy loss, even outperforming prior state-of-the-art methods like PLAID. The research provides theoretical guarantees for FDEs and demonstrates their effectiveness across various information retrieval datasets.

67 authors were involved in this research! This source is an academic paper titled "PaLM: Scaling Language Modeling with Pathways," authored by Aakanksha Chowdhery and numerous collaborators. It details the development and capabilities of PaLM (Pathways Language Model), a 540-billion parameter Transformer language model trained on 6144 TPU v4 chips using a new ML system called Pathways. The paper highlights PaLM's state-of-the-art performance in few-shot learning across various natural language tasks, including multilingual tasks and source code generation. Additionally, the authors provide analysis on bias, toxicity, and training data memorization, alongside a discussion of ethical considerations related to large language models. The document is hosted on arXiv, an open-access repository for scholarly articles, and includes submission history, full-text links, and citation tools. https://arxiv.org/abs/2204.02311

This paper introduces a multi-agent debate framework designed to enhance the factuality and reasoning capabilities of large language models (LLMs). The core idea involves multiple instances of an LLM proposing and critiquing solutions in iterative rounds until a consensus is reached. The authors demonstrate that this "society of minds" approach significantly improves performance across various tasks, including mathematical reasoning, strategic game play, and generating factually accurate biographies, by reducing errors and hallucinations often seen in single-model outputs. This method is directly applicable to existing black-box LLMs and can even lead to correct answers when individual models initially err, suggesting a powerful avenue for LLM self-improvement. Source: https://arxiv.org/pdf/2305.14325

Thus paper introduces PartitionedVC, an innovative external memory graph analytics framework designed to enhance the processing of large graphs that exceed main memory capacity, especially when utilizing SSDs. The core of PartitionedVC's improvement over existing systems like GraphChi lies in its use of a compressed sparse row (CSR) based graph storage for efficiently loading only active vertices, unlike shard-based frameworks that load entire graph segments regardless of activity. To overcome CSR's limitation with random updates, PartitionedVC employs a multi-log update mechanism, dedicating a separate log for each vertex interval to streamline update processing and eliminate costly external sorting. Furthermore, it incorporates an edge-log optimizer that proactively logs outgoing edges of likely active vertices, significantly reducing read amplification and overall performance bottlenecks associated with SSD page granular access.