Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：Demystifying Long Chain-of-Thought Reasoning in LLMs

Summary

This paper investigates how large language models (LLMs) achieve long chain-of-thought (CoT) reasoning, which involves extended, step-by-step thought processes for complex tasks. The authors explore the roles of supervised fine-tuning (SFT) and reinforcement learning (RL) in enabling this capability. Key findings highlight that while SFT on long CoT data improves performance and facilitates better RL, carefully designed reward functions are crucial for stable CoT length and enhanced reasoning. The study also examines the use of noisy web data for training and nuances in analyzing emergent reasoning behaviors during RL from base models. Ultimately, the research offers practical insights for optimizing training strategies to bolster sophisticated reasoning in LLMs.

本文探讨了大型语言模型（LLMs）如何实现长链式思维（CoT）推理，即在复杂任务中执行逐步、扩展的思考过程。作者研究了监督微调（SFT）和强化学习（RL）在提升这一能力中的作用。关键发现包括：虽然在长 CoT 数据上进行 SFT 可提高性能并优化 RL 训练，但精心设计的奖励函数对于稳定 CoT 长度和增强推理能力至关重要。此外，研究还分析了带噪声的网页数据用于训练的影响，以及在 RL 过程中基于基础模型解析涌现推理行为的细微差别。最终，该研究提供了优化训练策略的实用见解，以提升 LLMs 的高级推理能力。

原文链接：https://arxiv.org/abs/2502.03373

####🥚####彩####蛋####🥚####

本博客从24年10月2日开启，已更新半年，借助NotebookLM的能力进行论文解读，专注人工智能、大模型、机器人算法的论文，也有几十个听友。本人是一介书生，现计划建立微信群，平时可以聊聊技术，聊聊生活。希望博客可以继续更新下去！

进群添加微信小助手：seventy3_podcast

备注：小宇宙

####🥚####彩####蛋####🥚####

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills

Summary

The provided research paper introduces ASAP, a novel two-stage framework designed to bridge the gap between simulated and real-world physics for humanoid robots, enabling them to perform complex, agile movements. The first stage involves pre-training control policies in simulation using human motion data. The second stage deploys these policies in the real world to collect data and train a "delta action model" that learns to compensate for discrepancies in dynamics. This model is then integrated back into the simulator to fine-tune the control policies, allowing for more accurate and agile real-world execution. Experiments demonstrate that ASAP significantly improves the ability of humanoid robots to perform challenging tasks, outperforming existing methods in both simulated and real environments. The work highlights a promising direction for transferring skills learned in simulation to physical robots, ultimately leading to more versatile and capable humanoids.

该研究提出了 ASAP，一种创新的两阶段框架，旨在弥合仿真与现实物理之间的差距，使人形机器人能够执行复杂且灵活的运动。第一阶段在仿真环境中使用人类运动数据进行控制策略的预训练。第二阶段将这些策略部署到现实环境，采集数据并训练一个**“增量动作模型”（delta action model），用于补偿动力学差异。随后，该模型被集成回仿真环境，以微调控制策略**，从而实现更精准、灵活的现实世界执行。实验结果表明，ASAP 显著提升了人形机器人完成高难度任务的能力，无论在仿真还是现实环境中均优于现有方法。本研究为仿真训练迁移至现实机器人提供了一条有效途径，推动人形机器人向更多功能、更智能的方向发展。

原文链接：https://arxiv.org/abs/2502.01143

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：LLM-AutoDiff: Auto-Differentiate Any LLM Workflow

Summary

The provided research introduces LLM-AutoDiff, a novel framework for automating prompt engineering for complex Large Language Model workflows. This system extends gradient-based optimization to multi-step and cyclic LLM applications by treating textual inputs as trainable parameters. LLM-AutoDiff constructs a graph representing the workflow, enabling a "backward engine" LLM to generate feedback that guides iterative prompt improvements, even across functional nodes and repeated calls. The framework incorporates techniques like selective gradient computation and two-stage validation to enhance efficiency. Experimental results demonstrate that LLM-AutoDiff outperforms existing methods in accuracy and training cost across various tasks, offering a new paradigm for scaling and automating LLM deployments.

该研究提出了 LLM-AutoDiff，一个自动化提示工程（prompt engineering）的新框架，旨在优化复杂的大型语言模型（LLM）工作流。该系统通过将文本输入视为可训练参数，将基于梯度的优化方法扩展到多步和循环 LLM 应用。LLM-AutoDiff 构建了一个表示工作流的计算图，并利用**“反向引擎” LLM** 生成反馈，指导跨功能节点和重复调用的迭代提示优化。该框架还引入了选择性梯度计算和双阶段验证等技术，以提高优化效率。实验结果表明，LLM-AutoDiff 在多个任务上的准确性和训练成本方面均优于现有方法，为 LLM 部署的自动化和规模化提供了一种新范式。

原文链接：https://arxiv.org/abs/2501.16673

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：s1: Simple test-time scaling

Summary

This research explores improving language model reasoning through a technique called test-time scaling, where extra computation during inference enhances performance. The authors introduce s1K, a small, high-quality dataset of reasoning problems, and budget forcing, a method to control the model's computational effort at test time. By finetuning a language model on s1K and using budget forcing, they achieve strong results on math reasoning benchmarks, even surpassing previously reported methods while using significantly less training data. The work also analyzes different approaches to test-time scaling, finding sequential methods like budget forcing more effective than parallel ones like majority voting. Ultimately, this study demonstrates a sample-efficient way to boost reasoning through strategic test-time computation.

本研究探讨了通过测试时扩展（test-time scaling）提升语言模型推理能力的方法，即在推理阶段增加计算量以增强性能。作者提出了s1K——一个小型高质量的推理问题数据集，并引入了预算强制（budget forcing），一种在测试时控制模型计算资源的方法。通过在 s1K 上微调语言模型并应用预算强制，研究在数学推理基准上取得了优异成绩，甚至在训练数据大幅减少的情况下超越了此前的方法。此外，研究分析了不同的测试时扩展策略，发现顺序方法（如预算强制）比并行方法（如多数投票）更有效。最终，该研究证明了一种数据高效的方式，即通过策略性测试时计算来提升推理能力。

原文链接：https://arxiv.org/abs/2501.19393

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：Spurious Forgetting in Continual Learning of Language Models

Summary

This paper introduces the concept of spurious forgetting in large language models during continual learning, distinguishing it from actual knowledge loss and attributing it to the disruption of task alignment. The authors demonstrate through experiments and theoretical analysis that early training on new tasks can misalign the model, particularly in the bottom layers. To address this, they propose a Freezing strategy that keeps the initial layers unchanged, significantly improving performance in various continual learning scenarios like safety alignment and instruction tuning. Their findings highlight the importance of task alignment over pure knowledge retention and offer a practical method to mitigate performance degradation.

本文引入了大型语言模型在持续学习过程中出现的虚假遗忘概念，将其与实际的知识丧失区分开来，并将其归因于任务对齐的破坏。作者通过实验和理论分析表明，在新任务的早期训练阶段，模型（尤其是底层）可能会发生错位。为此，他们提出了一种冻结策略，即保持初始层不变，从而在安全对齐、指令微调等多种持续学习场景下显著提升模型性能。研究结果强调了任务对齐的重要性，相较于单纯的知识保留更为关键，并提供了一种实用的方法来缓解性能下降问题。

原文链接：https://arxiv.org/abs/2501.13453

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training

Summary

This paper explores the surprising parallels between learning-rate schedules used in large model training and theoretical performance bounds from convex optimization. It demonstrates that a simple learning-rate schedule with a constant phase followed by a linear cooldown mirrors the behavior predicted by theory, even for non-convex deep learning problems. Furthermore, the research shows how this theoretical understanding can be practically applied to improve learning-rate tuning for continued training and transfer optimal rates across different schedules, leading to tangible gains in model performance. The work provides theoretical justification for empirically successful scheduling techniques and suggests that principles from convex optimization offer valuable insights into the training of complex neural networks.

本文探讨了大型模型训练中使用的学习率调度与凸优化理论性能界限之间的惊人相似性。研究表明，一个简单的学习率调度方案——先保持恒定，然后线性降温——即使在非凸深度学习问题中，也能呈现出与理论预测相符的行为。此外，研究还展示了如何将这一理论理解实际应用于改进持续训练的学习率调优，并在不同的调度方案之间转移最优学习率，从而显著提升模型性能。本研究为经验上成功的调度技术提供了理论依据，并表明凸优化的原理可以为复杂神经网络的训练提供有价值的见解。

原文链接：https://arxiv.org/abs/2501.18965

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：TokenVerse: Versatile Multi-concept Personalization in Token Modulation Space

Summary

TokenVerse introduces a new method for multi-concept personalization in text-to-image generation. The technique extracts visual elements and attributes from single or multiple images using only text captions and a pre-trained diffusion model. By leveraging the modulation space within Diffusion Transformers, TokenVerse disentangles complex concepts like objects, poses, and lighting. This enables users to combine these learned concepts in novel ways to create customized images without needing additional supervision like masks. TokenVerse shows significant advantages over existing personalization techniques, providing greater flexibility and control for personalized content creation and storytelling. The paper presents quantitative and qualitative results demonstrating the effectiveness of the TokenVerse framework.

TokenVerse 提出了一个用于 文本到图像生成 的新方法，旨在实现多概念个性化。该技术通过仅使用文本描述和预训练的扩散模型，从单一或多个图像中提取视觉元素和属性。通过利用扩散变换器（Diffusion Transformers）中的调制空间，TokenVerse 解构了诸如物体、姿势和光照等复杂概念。

这种方法使用户能够以创新的方式将这些学习到的概念进行组合，从而创建个性化图像，而无需像遮罩（masks）之类的额外监督。与现有的个性化技术相比，TokenVerse 展现了显著的优势，提供了更大的灵活性和控制力，促进了个性化内容创作和叙事的实现。

论文通过定量和定性结果展示了 TokenVerse 框架的有效性，证明了其在个性化生成和故事创作中的潜力。

原文链接：https://arxiv.org/abs/2501.12224

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs

Summary

This research introduces TensorLLM, a novel framework for improving the reasoning abilities and compression of Large Language Models (LLMs) by focusing on the Multi-Head Attention (MHA) block. The method employs multi-head tensorisation and Tucker decomposition to denoise and compress MHA weights by enforcing a shared higher-dimensional subspace across multiple attention heads. Experiments demonstrate that TensorLLM enhances LLM reasoning capabilities across various benchmark datasets and architectures without requiring additional training. The framework can also be combined with existing techniques that denoise the feed-forward network (FFN) layers for further performance gains. The study validates the approach through ablation experiments and comparisons with other compression techniques, showing consistent improvements in accuracy and compression rates. The paper concludes by emphasizing the potential of TensorLLM as a versatile module for improving LLMs and suggesting future work on finding generalizable hyperparameter settings.

本研究提出了 TensorLLM，一种新颖的框架，通过聚焦于多头自注意力（MHA）块来提升大型语言模型（LLM）的推理能力和压缩效率。该方法采用多头张量化和Tucker 分解，通过在多个注意力头之间强制共享一个更高维度的子空间，来去噪和压缩 MHA 权重。

实验表明，TensorLLM 在不同的基准数据集和架构上提升了 LLM 的推理能力，而无需额外的训练。该框架还可以与现有的去噪前馈网络（FFN）层的技术结合，进一步提升性能。通过消融实验和与其他压缩技术的比较，研究验证了该方法的有效性，显示出在准确性和压缩率方面的持续改进。

论文最后强调了 TensorLLM 作为一种多功能模块，具有提升 LLM 性能的潜力，并提出了未来研究的方向，即寻找可以广泛应用的超参数设置。

原文链接：https://arxiv.org/abs/2501.15674

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning

Summary

The paper introduces MMOA-RAG, a novel approach to improve Retrieval-Augmented Generation (RAG) by framing it as a multi-agent reinforcement learning problem. It addresses the issue of independently optimized RAG components by treating each module (query rewriting, document retrieval, etc.) as an individual agent. MMOA-RAG uses multi-agent reinforcement learning to align each agent's goal with the overarching goal of generating accurate answers. Experiments on question-answering datasets demonstrate that MMOA-RAG outperforms existing methods by jointly optimizing the modules and addressing interdependencies. Ablation studies validate the contribution of each component, supporting MMOA-RAG's adaptability across datasets.

论文介绍了 MMOA-RAG，一种通过将 检索增强生成（RAG） 问题转化为多智能体强化学习问题的新方法。该方法解决了传统 RAG 中各模块（如查询重写、文档检索等）独立优化的问题，将每个模块视为一个独立的智能体。MMOA-RAG 采用多智能体强化学习，使每个智能体的目标与生成准确答案的总体目标对齐。

在问答数据集上的实验表明，MMOA-RAG 通过联合优化各个模块并解决模块之间的相互依赖，超越了现有方法。消融实验验证了每个组件的贡献，进一步支持了 MMOA-RAG 在不同数据集上的适应性。

原文链接：https://arxiv.org/abs/2501.15228

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：Docling: An Efficient Open-Source Toolkit for AI-driven Document Conversion

Summary

Docling is a new open-source toolkit for document conversion, designed to parse various document formats into a structured representation using AI models for layout analysis and table recognition. It aims to provide an efficient and customizable solution for tasks like document understanding and information extraction, and it supports local execution, integrations with frameworks like LangChain and LlamaIndex. The paper outlines Docling's design, architecture (including pipelines, parser backends, and the DoclingDocument data model), AI models, and performance benchmarks compared to other open-source tools. The toolkit's capabilities make it suitable for generative AI applications, data preparation, and knowledge extraction, with future work planned to include more models and an open-source quality evaluation framework. Docling has attracted significant community interest and is integrated into several open-source projects.

Docling 是一个全新的开源文档转换工具包，旨在通过使用 AI 模型进行布局分析和表格识别，将各种文档格式解析为结构化表示。它旨在为文档理解和信息提取等任务提供高效且可定制的解决方案，支持本地执行，并与像 LangChain 和 LlamaIndex 等框架进行集成。

本文概述了 Docling 的设计与架构，包括管道、解析器后端和 DoclingDocument 数据模型，介绍了所使用的 AI 模型及其与其他开源工具的性能基准对比。该工具包的功能使其非常适合用于生成型 AI 应用、数据准备和知识提取等任务，未来的工作将包括更多模型的引入以及一个开源的质量评估框架。

Docling 已吸引了大量社区关注，并已集成到多个开源项目中。

原文链接：https://www.arxiv.org/abs/2501.17887