thought

J-lens는 attribution graph보다는 훨씬 접근이 용이하지만,

 

feature 간의 causal structure를 보여주지 않는다는 단점이 있다.

 

생각이 흘러가는 경로를 보여주지 않는다.

 

(그 외에도 여러가지 한계점이 있다).

 

따라서, preliminary step으로써, 모델이 무슨 생각으로 이런 output을 내보냈는지 살펴볼 수는 있겠지만,

궁극적으로 나의 궁금증인,

"어떤 추론 과정을 거쳐서 답변을 내놓았는가?" 에 thorough한 답변을 얻으려면

attribution graph를 그리는 방법이 최선이다..!

---

여러 interpretability methods 간에 특징은 Related work section에도 잘 기술되어 있다.

Comparison to other interpretability tools. 

The J-lens is one of many techniques for reading out the contents of an activation vector, which differ in expressivity, cost, and mechanistic grounding.

 

Linear probes [75, 76, 23] are cheap but supervised: each probe measures one researcher-specified concept. They are also correlational, rather than causal, in the sense that a probe may recover information the model encodes but does not itself use [77, 78].

 

Sparse dictionary learning [30, 79, 25] is unsupervised and yields a linear decomposition into features, but training a dictionary is expensive, and each feature requires a further interpretation step via top-activating examples or automated description [80, 81].

 

Attribution graphs [69, 68, 71, 16] combine dictionary features with per-input linear attribution to produce circuit diagrams of specific computations. Such graphs can be used to answer which upstream features caused a given output in a given context, but do not reveal what concepts an activation vector is generally poised to verbalize.

 

At the most expressive end sit methods that produce free-text descriptions of an activation: patching it into a prompting template for the model itself to decode [82, 83] , training a supervised question-answering oracle [84, 85], or training a natural-language autoencoder to reconstruct activations through a text bottleneck [86]. These can articulate multi-token concepts and relationships the J-lens cannot, but at substantially higher cost, and with additional risk of confabulations that are not grounded in the model’s actual activations.

 

The J-lens sits near the cheap-and-grounded end of this spectrum: a single precomputed matrix multiply per layer, derived analytically from the model's Jacobian and applicable uniformly to activations, weights, and feature directions, at the cost of output limited to a ranked list of single tokens. We view it as complementary to the more expressive methods rather than competitive with them.

 

< Attribution graphs >

Transcoders find interpretable LLM feature circuits NeurIPS 2024

Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models ICLR 2025

Circuit Tracing: Revealing Computational Graphs in Language Models 2025

On the Biology of a Large Language Model 2025

---