Features

Probing

Analyze the internal representations of neural networks. supervised models trained to predict input properties from the representations, aiming to asses how much information about the property is encoded in them.

 

Although performance on the probing task is interpreted as evidence for the amount of information encoded in the representations, there exists a tension between the ability of the probe to evaluate the information encoded and the probe learning the task itself.

 

Feature directions can be found in LMs using linear classifiers (linear probes). Highly accurate probes indicate a correlation between input representations and labels, but do not provide evidence that the model is using the encoded information for its predictions.

Linear Representation Hypothesis

Features are encoded as linear subspaces of the representation space.  Recent work suggests the linearity of concepts in representation space is largely driven by the next-word-prediction training objective and inductive biases in gradient descent optimization.

The linear representation hypothesis and the geometry of large language models

https://arxiv.org/pdf/2311.03658

 

On the origins of linear representations in large language models

https://arxiv.org/pdf/2403.03867

Sparse Autoencoders

Neurons are polysemantic. Two main reasons can explain polysemanticity. Firstly, features can be represented as linear combinations of the standard basis vectors of the neuron space, not corresponding to the basis elements themselves. Therefore, each feature is represented across many individual neurons, which is known as distributed representations. 

 

Secondly, given the extensive capabilities and long-tail knowledge demonstrated by large language models, it has been hypothesized that models could encode more features than they have dimensions, a phenomenon called superposition.

 

A possible strategy to disentangle features in superposition involves finding an overcomplete feature basis via dictionary learning. Autoencoders with sparsity regularization, also known as sparse autoencoders (SAEs), can be used for dictionary learning by optimizing them to reconstruct internal representations of a neural network exhibiting superposition while simultaneously promoting feature sparsity. 

By inducing sparsity on the latent representation of SAE feature activations h(z) = ReLU(zWenc + b) ∈ R^m and setting m > d, we can approximate z as a sparse linear combination of the rows of the learned W_dec ∈ R^mxd dictionary, from which we can extract interpretable and monosemantic SAE features. Since the output weights of each SAE feature
interact linearly with the residual stream, we can measure their direct effect on the logits and their composition with later layers' component. 

Marks et al. (2024) proposes an adaptation of the causal model framework aiming to incorporate SAE features and errors as nodes of the computational graph. Using edge attribution patching, they recover sparse feature circuits providing more intuitive overviews of features driving model predictions.

 

Taking features out of superposition with sparse autoencoders.

https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition

 

Towards monosemanticity: Decomposing language models with dictionary learning

https://transformer-circuits.pub/2023/monosemantic-features/index.html

 

Sparse autoencoders find highly interpretable features in language models

https://arxiv.org/pdf/2309.08600

 

Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt

https://www.semanticscholar.org/paper/Dictionary-Learning-Improves-Patch-Free-Circuit-in-He-Ge/1301ed763095097ff424c668e16a265b3ae2f231

 

Sparse feature circuits: Discovering and editing interpretable causal graphs in language models.

https://arxiv.org/pdf/2403.19647

SAEs Evaluations

The goal of SAEs is to learn sparse reconstructions of representations. To assess the quality of a trained SAE in achieving this it is common to compute the Pareto frontier of two metrics on an evaluation set. 

 

These metrics are: • The L0 norm of the feature activations vector h(z), which measures how many features are “alive” given an input. • The loss recovered, which reflects the percentage of the original cross-entropy loss of the LM across a dataset when substituting the original representations with the SAE reconstructions.

 

Feature density histogram: Feature density is the proportion of tokens in a dataset where a SAE feature has a non-zero value. By looking at the distribution of feature densities we can distinguish if the SAE learnt features that are too dense (activate too often) or too sparse (activate too rarely). 

 

The degree of interpretability of sparse features can be estimated based on theire direct logit attribution and maximally activating examples. This process can be done manually or automated, using a LLM to produce natural language explanations of SAE features. Although recent studies have compared various SAE architectures, developing robust evaluation frameworks to compare between architectures remains a critical area for future research.

 

Towards monosemanticity: Decomposing language models with dictionary learning.

https://transformer-circuits.pub/2023/monosemantic-features/index.html

 

Towards principled evaluations of sparse autoencoders for interpretability and control

https://arxiv.org/pdf/2405.08366

 

Measuring progress in dictionary learning for language model interpretability with board game models

https://arxiv.org/pdf/2408.00113

Variants

Improving dictionary learning with gated sparse autoencoders

https://arxiv.org/pdf/2404.16014

 

Announcing neuronpedia: Platform for accelerating research into sparse autoencoders

https://www.alignmentforum.org/posts/BaEQoxHhWPrkinmxd/announcing-neuronpedia-platform-for-accelerating-research

 

Scaling and evaluating sparse autoencoders

https://arxiv.org/pdf/2406.04093v1

 

Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

https://arxiv.org/pdf/2407.14435

 

Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2.

https://arxiv.org/pdf/2408.05147