Sign up to save your podcasts
Or
Share In Case You Missed It: ARC 'Challenge' Is Not That Challenging
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
In Case You Missed It: ARC 'Challenge' Is Not That Challenging
đ€ Upvotes: 8 | cs.CL, cs.AI
Authors:
Ćukasz Borchmann
Title:
In Case You Missed It: ARC 'Challenge' Is Not That Challenging
Arxiv:
http://arxiv.org/abs/2412.17758v1
Abstract:
ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged. We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.
View all episodes
By Jingwen Liang, Gengyu Wangđ€ Upvotes: 8 | cs.CL, cs.AI
Authors:
Ćukasz Borchmann
Title:
In Case You Missed It: ARC 'Challenge' Is Not That Challenging
Arxiv:
http://arxiv.org/abs/2412.17758v1
Abstract:
ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged. We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.