Deep learning works extraordinarily well. And we still largely don't know why.

A new paper from Jamie Simon, Daniel Kunin, and 12 co-authors argues that a scientific theory of deep learning is emerging, and coins a name for the emerging field: learning mechanics.

We sat down with Jamie and Dan on Generally Intelligent to talk about what a physics of deep learning would actually look like, why now, and what's left to figure out.

00:03:05 Learning mechanics as the physics to mechanistic interpretability's biology

00:04:13 Why deep learning needs a theory

00:07:07 Why deep learning is uniquely hard to engineer

00:12:11 How a week in the woods became a paper

00:25:59 The barrier to theory isn't opacity, but complexity

00:36:26 Deep learning's first gas law

00:47:22 Why more particles makes the problem easier

00:56:22 The discretization hypothesis

01:01:50 The strongest signal that a compact theory exists

01:05:07 The Platonic Representation Hypothesis

01:15:41 Why learning mechanics and mech interp need each other

01:25:29 Theory as safety infrastructure

Read the paper

Transcript and links

Learning Mechanics website

Full transcript: https://imbueai.substack.com/p/geoffrey-litt

Generally Intelligent is a podcast by ⁠⁠Imbue⁠⁠, a research company building toward a future where AI agents are open and accountable to their users, so people have more power in the digital world.

• Website
• Substack
• ⁠X⁠
• LinkedIn
• YouTube