Circuits

Circuit discovery

Recent studies in MI aim to uncover the existence of circuits, which are a subset of model components (subgraphs) interacting together to solve a task. Activation patching, logit attribution, and attention pattern analysis are common techniques for circuit discovery.

 

However, circuit discovery also requires identifying important interactions between components. For this purpose, edge patching exploits the fact that every model component input is the sum of the output of previous components in its residual stream, and considers edges directly connecting pairs of model components' nodes. Path patching generalizes the edge patching approach to multiple edges, allowing for a more fine-grained analysis of circuits.

---

INTERPRETABILITY IN THE WILD: A CIRCUIT FOR INDIRECT OBJECT IDENTIFICATION IN GPT-2 SMALL

https://openreview.net/pdf?id=NpsVSN6o4ul

 

A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis

https://arxiv.org/pdf/2305.15054

 

A circuit for Python docstrings in a 4-layer attention-only transformer

https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only

 

Dissecting Recall of Factual Associations in Auto-Regressive Language Models

https://aclanthology.org/2023.emnlp-main.751.pdf

 

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

https://papers.nips.cc/paper_files/paper/2023/file/efbba7719cc5172d175240f24be11280-Paper-Conference.pdf

 

LOCALIZING MODEL BEHAVIOR WITH PATH PATCHING

https://arxiv.org/pdf/2304.05969

Limitations & Overcoming

IS THIS THE SUBSPACE YOU ARE LOOKING FOR? AN INTERPRETABILITY ILLUSION FOR SUBSPACE ACTIVATION PATCHING

https://openreview.net/pdf?id=Ebt7JgMHv1

 

A reply to makelov et al. (2023)’s "interpretability illusion" arguments

https://arxiv.org/pdf/2401.12631

 

Explorations of Self-Repair in Language Models

https://arxiv.org/pdf/2402.15390

 

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

https://arxiv.org/pdf/2307.09458

 

Towards automated circuit discovery for mechanistic interpretability

https://papers.nips.cc/paper_files/paper/2023/file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf

 

Sparse feature circuits: Discovering and editing interpretable causal graphs in language models

https://arxiv.org/pdf/2403.19647

 

Information Flow Routes: Automatically Interpreting Language Models at Scale

https://arxiv.org/pdf/2403.00824

 

AtP∗ : An efficient and scalable method for localizing LLM behaviour to components

https://arxiv.org/pdf/2403.00745