[Draft] Testing SCM Faithfulness in LLM Internals

아직 확신이 없음.

paper 읽고 생각을 정리해야 한다.

대충 GPT pro와 discussion으로 정리한 내용

---

* Research Question

Do LLMs internally implement the causal graph described in a prompt, or do they answer causal questions using surface-level correlations?

* Setup

Use synthetic SCMs with 3-8 variables. Generate many natural-language descriptions with randomized variable names, randomized graph structures, and controlled observational/interventional queries.

* Models

Use open-weight models where we can access activations: small models first, then instruction-tuned models.

* Methods

- Linear probes for causal variable values.

- Sparse autoencoders for interpretable feature discovery.

- Activation patching to test causal use.

- Interchange interventions to test alignment with causal variables.

- Feature-level graph discovery via interventions.

- Behavioral comparison against standard causal benchmarks.

* Metrics

- Answer accuracy.

- Interventional query accuracy.

- Counterfactual query accuracy.

- Interchange intervention accuracy.

- Structural Hamming Distance between recovered internal feature graph and gold DAG.

- Average treatment effect error

- OOD generalization under variable renaming, graph reparameterization, and misleading commonsense labels.

* Critical controls

Use graph pairs with the same observational distribution but different interventional distributions. If the model follows correlation, it should fail; if it represents the SCM, it should distinguish them.

 

Also include misleading variables names and abstract anmes. This tests whether the model is relying on prior world knowledge or on the causal structure described in context.

* Claim

We identify internal features corresponding to causal variables in a prompted SCM. Intervening on these features produces downstream activation and output changes that match the SCM's do-intervention predictions. This holds across paraphrases, variable renamings, and out-of-distribution graph structures, and fails in predictable ways for collider/confounder cases.

 

We study whether LLMs perform causal inference by implementing internal causal abstractions. Using synthetic and semi-synthetic SCMs, We identify model-internal causal variables and test whether interventions on these variables reproduce the predictions of the external causal graph.

 

We ask following question, can mechanistic interpretability tell us when LLM causal reasoning is faithful, when it is shorcut-based, and how to intervene on it?