TLDR;

In previous work, we found a problematic form a feature splitting called "feature absorption" when analyzing Gemma Scope SAEs. We hypothesized that this was due to SAEs struggling to separate co-occurrence between features, but we did not prove this. In this post, we set up toy models where we can explicitly control feature representations and co-occurrence rates and show the following:

• Feature absorption happens when features co-occur.
• If co-occurring feature magnitudes vary relative to each other, we observe "partial absorption", where a latent tracking a main feature sometimes fires weakly instead of not firing at all, but sometimes does fully not fire.
• Feature absorption happens even with imperfect co-occurrence, depending on the strength of the sparsity penalty.
• Tying the SAE encoder and decoder weights together solves feature absorption.

All code for this post can be seen in this Colab notebook.

The rest of this post will assume [...]

---

Outline:

(00:06) TLDR;

(01:18) What is feature absorption?

(02:15) How is this different than traditional feature splitting?

(03:47) Why does absorption happen?

(04:17) How big of a problem is this, really?

(05:04) Toy Models of Feature Absorption Setup

(05:29) Non-superposition setup

(07:12) Superposition setup

(07:53) Perfect Reconstruction with Independent Features

(08:39) Feature co-occurrence causes absorption

(10:33) Magnitude variance causes partial absorption

(12:35) Why does partial absorption happen?

(13:13) Imperfect co-occurrence can still lead to absorption depending on L1 penalty

(16:03) Tying the SAE encoder and decoder weights solves feature absorption

(17:08) Absorption in superposition

(19:13) Tying the encoder and decoder weights still solves feature absorption in superposition.

(19:53) Future work

The original text contained 10 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.