AI Post Transformers

This August 2025 paper introduces Self-Search Reinforcement Learning (SSRL), a novel method that enables Large Language Models (LLMs) to access and utilize their internal knowledge for search-driven tasks, bypassing the need for external search engines like Google or Bing. The research explores how repeated sampling can enhance an LLM's intrinsic search capabilities and investigates the impact of various prompting strategies and training methodologies, including the benefits of information masking and format-based rewards. The paper demonstrates that SSRL-trained models can effectively generalize to real-world search scenarios while often outperforming methods that rely on external search APIs, suggesting LLMs can function as powerful internal knowledge bases for complex queries. Source: https://arxiv.org/pdf/2508.10874

This paper February 2025 paper introduces AiSAQ (All-in-Storage ANNS with Product Quantization), a novel method designed for Approximate Nearest Neighbor Search (ANNS) that significantly reduces DRAM (Dynamic Random-Access Memory) usage. Unlike traditional methods like DiskANN, which store compressed vectors in DRAM, AiSAQ offloads these to SSD (Solid-State Drive) storage, allowing for near-zero memory footprint even with billion-scale datasets. The paper details how this approach maintains high search performance while drastically lowering index load times and enabling rapid switching between large datasets, making it particularly beneficial for applications like Retrieval-Augmented Generation (RAG) in Large Language Models (LLMs) and multi-server environments. Experiments demonstrate AiSAQ's efficiency in terms of memory, latency, and cost-effectiveness for large-scale information retrieval. Source: February 2025 https://arxiv.org/pdf/2404.06004

This February 2025 paper introduces ELMo-Tune-V2, a novel framework that leverages Large Language Models (LLMs) to fully automate the optimization of Log-Structured Merge-tree-based Key-Value Stores (LSM-KVS). Unlike previous methods that rely on human experts or limited automated tuning, ELMo-Tune-V2 integrates LLMs for self-navigated workload characterization, automatic tuning across a broad parameter space, and real-time dynamic configuration adjustments. The framework demonstrates significant performance improvements for popular LSM-KVS systems like RocksDB, addressing the complex interplay of hardware, resource limits, and evolving workloads. ELMo-Tune-V2 achieves this through innovations in LLM-based workload synthesis, feedback-driven iterative fine-tuning, and real-time adaptive tuning, showcasing the potential of LLMs in solving complex data system optimization challenges. Source: https://arxiv.org/pdf/2502.17606

This February 2025 paper introduces fMoE, a novel fine-grained expert offloading system designed to optimize the serving efficiency of Mixture-of-Experts (MoE) Large Language Models (LLMs). The paper highlights the memory inefficiency of current MoE-based LLMs during inference due to inactive experts residing in GPU memory and the limitations of existing coarse-grained offloading solutions that struggle with latency-memory trade-offs. fMoE addresses these challenges by tracking iteration-level expert probability distributions through "expert maps" and leveraging input semantic embeddings to intelligently guide expert prefetching, caching, and offloading decisions. Experiments show that fMoE significantly reduces inference latency and improves expert hit rates compared to state-of-the-art methods. Source: https://arxiv.org/html/2502.05370v1

This is a huge review of 13 different sources on advancements in GPU-accelerated computing, focusing on data access, memory management, and performance optimization for large datasets. Several sources highlight NVIDIA's initiatives like GPUDirect Storage and the AI Data Platform, which streamline data transfer directly between storage and GPUs, reducing CPU bottlenecks. Conversely, other documents analyze AMD's efforts with ROCm, acknowledging its rapid software stack improvements but also pointing out challenges like lack of comprehensive Python support and the need for increased R&D investment to compete with NVIDIA's established CUDA ecosystem. Concepts such as GPU-orchestrated memory tiering and novel I/O primitives are presented as solutions to overcome limitations in GPU memory capacity and PCIe bandwidth, enabling more efficient processing of extensive data analytics and AI workloads. Source 1: GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture https://arxiv.org/pdf/2203.04910 Source 2: Vortex: Overcoming Memory Capacity Limitations in GPU-Accelerated Large-Scale Data Analytics https://arxiv.org/pdf/2502.09541 Source 3: GPU as Data Access Engines https://files.futurememorystorage.com/proceedings/2024/20240808_NETC-301-1_Newburn.pdf Source 4: Performance Analysis of Different IO Methods between GPU Memory and Storage https://www.tkl.iis.u-tokyo.ac.jp/new/uploads/publication_file/file/1051/6C-03.pdf Source 5: GDS cuFile API Reference - https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html Source 6: AMD 2.0 – New Sense of Urgency | MI450X Chance to Beat Nvidia | Nvidia’s New Moat Rapid Improvements, Developers First Approach, Low AMD AI Software Engineer Pay, Python DSL, UALink Disaster, MI325x, MI355x, MI430X UL4, MI450X Architecture, IF64/IF128, Flexible IO, UALink, IFoE https://semianalysis.com/2025/04/23/amd-2-0-new-sense-of-urgency-mi450x-chance-to-beat-nvidia-nvidias-new-moat/ Source 7: Accelerating and Securing GPU Accesses to Large Datasets https://www.nvidia.com/en-us/on-demand/session/gtc24-s62559/ Source 8: GMT: GPU Orchestrated Memory Tiering for the Big Data Era https://dl.acm.org/doi/10.1145/3620666.3651353 Source 9: GPUDirect Storage https://docs.nvidia.com/gpudirect-storage/ Source 10: GPUDirect Storage: A Direct Path Between Storage and GPU Memory https://developer.nvidia.com/blog/gpudirect-storage/ Source 11: Introducing ROCm-DS: GPU-Accelerated Data Science for AMD Instinct™ GPUs https://rocm.blogs.amd.com/software-tools-optimization/introducing-rocm-ds-revolutionizing-data-processing-with-amd-instinct-gpus/README.html Source 12: NVIDIA and Storage Industry Leaders Unveil New Class of Enterprise Infrastructure for the Age of AI https://nvidianews.nvidia.com/news/nvidia-and-storage-industry-leaders-unveil-new-class-of-enterprise-infrastructure-for-the-age-of-ai Source 13: Why is CUDA so much faster than ROCm? https://www.reddit.com/r/MachineLearning/comments/1fa8vq5/d_why_is_cuda_so_much_faster_than_rocm/

We review Colossal-AI's NVMe offload functionality, designed to overcome GPU memory limitations when training large-scale models by transferring optimizer states to NVMe disks. It highlights the TensorNVMe library, which facilitates this process and is compatible with various disk types, though NVMe SSDs are recommended for optimal performance. The text further explains the pipelined optimization process that overlaps computation and I/O, demonstrating its usage with CPUAdam and HybridAdam optimizers. Practical examples using GPT models illustrate the memory savings achieved through NVMe offloading for both CPU and Gemini-backed training. Finally, an API reference provides detailed information on the HybridAdam and CPUAdam classes and their parameters. Source: https://colossalai.org/docs/features/nvme_offload/

This reviews the IETF Parallel Network File System (pNFS), an extension to NFS that separates file metadata from data storage. Specifically, "RFC 8435" introduces the initial flexible file layout type, enabling pNFS to utilize existing protocols and storage devices with limited metadata server interaction, incorporating client-side mirroring for data replication. Then we review, "Parallel NFS (pNFS) Flexible File Layout Version 2," details an update to this layout, enhancing data protection with additional methods like the Mojette algorithm, and refining the management of tightly and loosely coupled storage device models, including how client I/O errors and layout usage statistics are reported and managed. Both sources discuss the mechanisms for recalling layouts, client fencing, and the security considerations inherent in these distributed file system architectures.Sources: https://www.ietf.org/archive/id/draft-ietf-nfsv4-layoutwcc-02.html - Add LAYOUT_WCC to NFSv4.2's Flex File Layout Typehttps://www.ietf.org/archive/id/draft-haynes-nfsv4-flexfiles-v2-00.html - Parallel NFS (pNFS) Flexible File Layout Version 2https://datatracker.ietf.org/doc/rfc8435/ - Parallel NFS (pNFS) Flexible File LayoutRFC 8435

A video transcript is used to review the PostgreSQL Development Conference (PGConf.dev 2025) presentation titled "Scaling Postgres to the Next Level at OpenAI." The speaker, Bhan from OpenAI, outlines their extensive experience scaling PostgreSQL to support critical, read-heavy workloads without traditional sharding, achieving millions of queries per second. He discusses optimizations implemented, such as reducing primary database load, improving query efficiency, and mitigating single points of failure. The talk also addresses past outages at OpenAI related to PostgreSQL, lessons learned, and areas where PostgreSQL could improve, like observability and schema change management. Source: https://youtu.be/Ni1SGhNu-Q4?si=CoHnwn7ccArykBAY

This 2022 paper is a reminder of issues with mmap() for databases. Yet many Vector Databases today rely on mmap(). This academic paper critically evaluates the use of memory-mapped file I/O (mmap) in Database Management Systems (DBMSs), arguing against its perceived benefits over traditional buffer pool implementations. The authors explain that while mmap appears to simplify file I/O by letting the Operating System (OS) handle data movement between storage and memory, it introduces significant correctness and performance issues. They detail problems concerning transactional safety, I/O stalls, error handling, and performance bottlenecks like TLB shootdowns, illustrating these points with experimental analysis. The paper concludes by advising against mmap for most DBMS applications, especially those requiring high throughput or transactional safety. Source: https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

The provided source introduces Mem0 and Mem0g, two novel memory architectures designed to enhance Large Language Models (LLMs) by overcoming their inherent context window limitations and improving long-term conversational coherence. Mem0 focuses on dynamically extracting, consolidating, and retrieving salient information from conversations in natural language text, while Mem0g augments this with graph-based memory representations to capture complex relational structures. The research evaluates these systems against various baselines, including established memory-augmented systems, Retrieval-Augmented Generation (RAG) approaches, and proprietary models, demonstrating superior performance in accuracy across different question types (single-hop, multi-hop, temporal, and open-domain). Furthermore, Mem0 and Mem0g significantly reduce computational overhead and latency compared to full-context processing, highlighting their practical viability for production-ready AI agents requiring persistent and efficient memory. The findings underscore the critical role of structured and dynamic memory mechanisms for enabling more reliable and effective LLM-driven interactions over extended periods. Source: https://arxiv.org/pdf/2504.19413