This post is a summary of our paper A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task (ACL 2024). While we wrote and released the paper a couple of months ago, we have done a bad job promoting it so far. As a result, we’re writing up a summary of our results here to reinvigorate interest in our work and hopefully find some collaborators for follow-up projects. If you’re interested in the results we describe in this post, please see the paper for more details.

TL;DR - We train transformer models to find the path from the root of a tree to a given leaf (given an edge list of the tree). We use standard techniques from mechanistic interpretability to figure out how our model performs this task. We found circuits that involve backward chaining - the first layer attends to [...]

---

Outline:

(01:52) Motivation and The Task

(05:18) Backward Chaining with Deduction Heads

(08:57) Register Tokens and Path Merging

(10:38) Final Heuristic

(12:09) Tuned Lens Visualization

(12:42) Takeaways and Limitations

(15:37) Future work

(16:26) Acknowledgments

---

First published:

Source:

Narrated by TYPE III AUDIO.