Seventy3：借助NotebookLM的能力进行论文解读，专注人工智能、大模型、机器人算法方向，让大家跟着AI一起进步。

进群添加小助手微信：seventy3_podcast

备注：小宇宙

今天的主题是：SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

Summary

The provided text introduces SWE-Lancer, a new benchmark designed to evaluate AI models on real-world freelance software engineering tasks sourced from Upwork, with a total payout value of $1 million. This benchmark includes both independent coding tasks and managerial tasks of selecting the best technical proposals. Unlike previous benchmarks that often rely on unit tests, SWE-Lancer uses end-to-end tests verified by experienced engineers and assesses managerial decisions against real hiring manager choices. The study evaluates the performance of several frontier AI models on this benchmark, finding that significant challenges remain in achieving high success rates on these practical software engineering problems, despite advancements in the field. The authors have also open-sourced a portion of the benchmark to encourage further research into the economic impact and capabilities of AI in software development.

该文本介绍了 SWE-Lancer，这是一个全新的基准测试，旨在评估AI模型在真实自由职业软件工程任务中的表现，这些任务均来自Upwork，总奖金价值达100万美元。该基准涵盖了独立编码任务以及需要做出技术提案选择的管理类任务。与以往主要依赖单元测试的基准不同，SWE-Lancer采用了由经验丰富工程师验证的端到端测试，并将AI的管理类决策与真实招聘经理的选择进行对比评估。

研究对多个前沿AI模型在该基准上的表现进行了测试，发现尽管AI在软件工程领域已有显著进展，但在应对这类实际工程问题时，仍面临不少挑战，成功率有待提高。为了推动该领域研究，作者还开源了部分基准内容，以鼓励对AI在软件开发中经济影响和能力的深入探索。

原文链接：https://arxiv.org/abs/2502.12115