We at Epoch AI have recently released a new composite AI capability index called the Epoch Capabilities Index (ECI), based on nearly 40 underlying benchmarks.

Some key features...

• Saturation-proof: ECI "stitches" benchmarks together, to enable comparisons even as individual benchmarks become saturated.
• Global comparisons: Models can be compared, even if they were never evaluated on the same benchmarks.
• Difficulty-based task weighting: ECI uses a simple statistical model (similar to those used in Item Response Theory) under which models deemed more capable if they score well on difficult benchmarks, and benchmarks are deemed more difficult if capable models are unable to score highly on them. 

ECI will allow us to track trends in capabilities over longer spans of time, potentially revealing changes in the pace of progress. It will also improve other analyses that would otherwise depend on a single benchmark for comparison.

You can find more details about [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.