Parameter vs. Activation Decomposition: Traditional interpretability methods like Sparse Autoencoders (SAEs) focus on analyzing activations, while parameter decomposition focuses on understanding the parameters themselves - the actual "algorithm" of the neural network.

No "True" Decomposition: None of the decompositions (whether sparse dictionary learning or parameter decomposition) are objectively "right" because they're all attempting to discretize a fundamentally continuous object, inevitably introducing approximations.

Tradeoff in Interpretability: There's a balance between reconstruction loss and causal importance - as you decompose networks more, reconstruction loss may worsen, but interpretability might improve up to a certain point.

Potential Unlearning Applications: Parameter decomposition may make unlearning more straightforward than with SAEs because researchers are already working in parameter space and can directly modify vectors that perform specific functions.

Function Detection vs. Input Direction: A function like "deception" might manifest in many different input directions that SAEs struggle to identify as a single concept, while parameter decomposition might better isolate such functionality.

Knowledge Extraction Goal: A key aim is to extract knowledge from models by understanding how they "think," especially for tasks where models demonstrate superhuman capabilities.