AI Post Transformers

This July 2025 research paper explores In-Context Learning (ICL) in Large Language Models (LLMs), which is the striking ability of these models to learn new patterns from examples given in a prompt without explicit weight updates during inference. The authors hypothesize and demonstrate through theory and experimentation that the combination of a self-attention layer and a Multi-Layer Perceptron (MLP) within the transformer architecture allows the context to implicitly modify the MLP's weights. They generalize this concept with the notion of a contextual block and provide a formula showing that the effect of the context is equivalent to a low-rank weight update of the neural network's first layer. This implicit process, they argue, acts as a form of implicit learning dynamics similar to gradient descent, where tokens consumed sequentially drive the weight adjustments. The findings suggest that ICL is rooted in how regular neural networks can transfer input modifications to their weight structure, rather than solely being about the self-attention mechanism. Source: https://arxiv.org/pdf/2507.16003

The October 2025 academic paper introduces a novel imperceptible jailbreaking attack against Large Language Models (LLMs) that exploits Unicode variation selectors, which are invisible characters. Unlike previous jailbreaking methods that rely on visible text modifications, this technique appends invisible variation selectors to malicious questions, visually preserving the original prompt while altering the LLM's tokenization to bypass safety alignment. The authors propose a chain-of-search pipeline to optimize these adversarial suffixes, achieving high attack success rates against four aligned LLMs and demonstrating generalization to prompt injection attacks. Through analysis of attention scores and embedding differences, the study confirms that the invisible suffixes successfully redirect the model's focus away from harmful content to produce unsafe outputs. Source: https://arxiv.org/pdf/2510.05025

The October 2025 paper introduces LongCodeZip, a novel, training-free, and model-agnostic framework designed for compressing long code contexts to improve the efficiency and capability of Code Large Language Models (LLMs). The core problem addressed is that long code contexts lead to high API costs, increased latency, and model difficulty in identifying relevant information due to the structured nature of code. LongCodeZip utilizes a two-stage hierarchical approach: coarse-grained compression selects the most relevant functions based on conditional perplexity (approximated mutual information), followed by fine-grained compression that further prunes code within these functions into semantically coherent blocks using perplexity-based chunking and a knapsack optimization to maximize information density. Evaluations across code completion, summarization, and question answering tasks demonstrate that LongCodeZip achieves up to a 5.6x compression ratio while consistently outperforming existing compression and retrieval-augmented generation (RAG) baselines, even when utilizing a smaller compression model. Source: https://arxiv.org/pdf/2510.00446

The September 2025 paper introduces MotionRAG, a novel retrieval-augmented framework designed to enhance motion realism in image-to-video generation. The central challenge addressed is the difficulty diffusion models face in generating videos with physically plausible and coherent motion. MotionRAG solves this by using a retrieval pipeline to adapt high-level motion priors from relevant reference videos through its core component, the Context-Aware Motion Adaptation (CAMA) module. This approach formulates motion transfer as an in-context learning problem, allowing the system to generalize to new domains with negligible computational overhead and without requiring fine-tuning of the base generation models. Experimental results demonstrate that MotionRAG significantly improves motion quality across various state-of-the-art models and datasets, including specialized domains, by effectively guiding video generation with real-world motion patterns. Source: https://arxiv.org/pdf/2509.26391

The provided text is an excerpt from a technical evaluation report conducted by the Center for AI Standards and Innovation (CAISI), housed within the National Institute of Standards and Technology (NIST), in September 2025. This report systematically compares three DeepSeek AI models against four U.S. reference models, including OpenAI’s GPT-5 and Anthropic’s Opus 4, across 19 benchmarks. The evaluation focuses on several critical areas, revealing that DeepSeek models generally lag U.S. models in performance, particularly in cyber and software engineering tasks, while also being more expensive to operate and significantly less robust against security threats like agent hijacking and jailbreaking attacks. Furthermore, the analysis determined that the DeepSeek models exhibit alignment with Chinese Communist Party (CCP) censorship narratives in both English and Chinese queries. The document also includes data on model adoption trends, noting the rapid increase in the use of certain PRC models like DeepSeek. Source: https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf

The provided sources center on the Open Neural Network Exchange (ONNX) format and its inference engine, ONNX Runtime, highlighting their role in enabling high-performance, cross-platform machine learning deployment. Several sources detail the architectural benefits of ONNX Runtime, such as enabling AI inference in Java systems without Python dependencies and facilitating hardware acceleration across various chips like NVIDIA GPUs and Arm processors. One critical source introduces OODTE, a differential testing tool used to assess the functional correctness of the ONNX Optimizer, revealing multiple bugs and accuracy deviations in optimized models. Finally, a practical example from Firefox AI demonstrates switching from the WebAssembly (WASM) version to the native C++ ONNX Runtime for a significant speed increase in local AI features.Sources:https://en.wikipedia.org/wiki/Open_Neural_Network_Exchangehttps://github.com/onnx/onnx/blob/main/docs/Overview.mdhttps://github.com/onnx/optimizerhttps://github.com/onnx/onnx/blob/main/docs/IR.mdhttps://blog.stackademic.com/onnx-open-neural-network-exchange-29f39a84c5f2https://developer.nvidia.com/blog/end-to-end-ai-for-pcs-onnx-runtime-and-optimization/https://developer.arm.com/ai/kleidi-librarieshttps://newsroom.arm.com/blog/arm-microsoft-kleidiai-onnx-runtimehttps://hackernoon.com/mobile-ai-with-onnx-runtime-how-to-build-real-time-noise-suppression-that-workshttps://blog.mozilla.org/en/firefox/firefox-ai/speeding-up-firefox-local-ai-runtime/https://www.infoq.com/articles/onnx-ai-inference-with-java/https://arxiv.org/pdf/2202.06929https://arxiv.org/html/2505.01892v1

The October 2025 paper introduces Paris, a novel open-weight diffusion model for text-to-image generation that was trained using a completely decentralized methodology without requiring any communication between its expert components. This approach overcomes the need for expensive, specialized hardware clusters and synchronized gradient updates, which are typically required for training large-scale models like Stable Diffusion. The system functions by partitioning the training data into semantically distinct clusters, allowing eight expert models to train in isolation, with a separate, lightweight router network dynamically selecting the most appropriate expert(s) during inference. Empirical results demonstrate that Paris achieves competitive generation quality while using substantially fewer computational resources and training data compared to prior decentralized benchmarks, making large generative AI models more accessible on heterogeneous, fragmented compute infrastructure. Source: https://arxiv.org/pdf/2510.03434

The October 2025 academic paper introduces RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a novel reinforcement learning (RL) method designed to improve the safety and robustness of large reasoning models (LRMs). The core problem addressed is the brittleness of LRMs, which are easily biased by flawed chain-of-thought (CoT) reasoning injected into their thought process, leading to unsafe or overly cautious responses. RECAP addresses this by explicitly training models on a mixture of standard prompts and counter-aligned CoT prefills—forcing the model to override unsafe reasoning for harmful queries or overly conservative refusals for benign ones to achieve a high reward. Experimental results show that RECAP substantially enhances safety, reduces overrefusal, and preserves core reasoning capabilities, leading to more frequent self-reflection in the models and persistent robustness against adaptive adversarial attacks. The method integrates easily with existing RL-from-human-feedback (RLHF) frameworks without incurring additional training costs. Source: https://arxiv.org/pdf/2510.00938

The October 2025 paper introduces the Reactive Transformer (RxT), a novel neural network architecture designed by Adam Filipek and Reactive AI to overcome the scaling and latency issues of current Large Language Models (LLMs) in long-form conversations. Unlike traditional stateless LLMs, which suffer from quadratic computational complexity by reprocessing the entire conversation history, RxT adopts an event-driven, stateful paradigm. The core innovation is an integrated, fixed-size Short-Term Memory (STM) system and an asynchronous operational cycle that decouples the fast response generation from the computationally intensive memory update, leading to linear scaling of total conversational cost. Experimental results on synthetic data demonstrate that RxT models, even smaller ones, significantly outperform comparable stateless LLMs in perplexity and conversational coherence while maintaining constant, low inference latency, validating the efficiency and design of the architecture and its four-stage training curriculum. Source: https://arxiv.org/pdf/2510.03561 https://rxai.dev

The September 2025 paper introduces ReasoningBank, a novel memory framework designed to enhance Large Language Model (LLM) agents by distilling and utilizing abstract reasoning patterns from both successful and failed task trajectories. Unlike previous approaches that focus only on raw interactions or successful workflows, ReasoningBank stores structured memory items—including titles, descriptions, and content—that capture generalizable strategies and lessons learned from mistakes. This framework is combined with Memory-Aware Test-Time Scaling (MaTTS), which uses memory to guide efficient exploration, creating a positive feedback loop where diverse experiences generate stronger memories, leading to consistently improved success rates and reduced steps across various complex agent tasks like web browsing and software engineering. Source: https://arxiv.org/pdf/2509.25140