Activation Patching

Activation Patching (Interchange Interventions)

We can view the computations of a Transformer-based LM as a causal model, and use causality tools to shed light on the contribution to the prediction of each model component across different positions. 

 

Causal Abstractions of Neural Networks

https://proceedings.neurips.cc/paper_files/paper/2021/file/4f5c422f4d49a5a807eda27434231040-Paper.pdf

 

The Hydra Effect: Emergent Self-repair in Language Model Computations

https://arxiv.org/pdf/2307.15771

 

How to use and interpret activation patching

https://arxiv.org/pdf/2404.15255

 

Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

https://arxiv.org/pdf/2303.02536