Residual-Controlled SAE Diffing for LLM Failure Discovery

Tech Stack

Algorithms

• Residual Error Modeling
• Residual-Based Failure Grouping
• Sparse Autoencoder (SAE) Diffing
• Latent Feature Attribution

Motivation

The key motivation is to move from ad hoc error anecdotes to systematic, reproducible failure taxonomies that can improve stress-testing, model diagnostics, and practical deployment guardrails.

Problem

Automatic failure discovery is still the bottleneck in LLM error-teaching pipelines. Prior approaches often depend on predefined labels and single-pattern checks, so they miss novel failures and provide weak criteria for comparing full discovered pattern sets.

Approach

The method follows a three-stage pipeline: (1) residual control using logistic regression over shallow question features to estimate expected error, then splitting high-residual (surprising) vs low-residual cohorts; (2) SAE diffing over Gemma 2 9B layer-20 hidden states with Welch t-tests across 16,384 latent dimensions; and (3) set-level evaluation using coverage, concentration, AUC, F1, and distinctness (1 - average Jaccard).

Methodology

• Residual control: train logistic regression on shallow features (for example text length and numeral count), then compute residual = actual error - predicted probability.
• Cohort split: isolate high-residual errors (top quartile, Q75 = 0.506) and low-residual errors (bottom quartile, Q25 = 0.417).
• SAE diffing: encode layer-20 hidden states with SAELens SAE (d_sae = 16,384), then run Welch t-tests to rank latent dimensions by effect size V'.
• Set-level scoring: evaluate the discovered pattern set with coverage, concentration, AUC, F1, and distinctness to measure utility and redundancy at set level.

Evaluation Plan

• Data and model setting: Gemma 2 9B on MMLU (57 subjects, 7,851 instances), with SAE analysis at layer_20 (width_16k).
• Primary metrics: coverage, concentration, AUC, F1, and distinctness (1 - average Jaccard), interpreted as whole-set quality signals rather than single-latent performance.
• Pattern diagnostics: use subject-level heatmaps and top-latent effect sizes to inspect where surprising failures concentrate.

Outcome

The end-to-end Stage 2 pipeline runs reproducibly on MMLU and produces interpretable artifacts, including subject-level latent heatmaps and set-level metrics. The run shows very high coverage (1.0) and usable concentration (0.767), but near-random ranking quality (AUC 0.498) and high redundancy (Jaccard 0.600), indicating strong observability with clear room for deduplication and ranking improvement.

Current Status

• Completed: end-to-end residual -> SAE -> set-metric pipeline with consolidated report artifacts.
• Completed: reproducible quantitative outputs for one full run (error rate 76.7%, high-residual coverage 32.6%, 779 candidate patterns).
• Completed: integrated interpretation views for Plot 2 (latent delta heatmap) and Plot 3 (set-level metrics radar + cards).

Evaluation & Metrics

• Benchmark profile: MMLU (57 subjects, 7,851 instances) with Gemma 2 9B; observed error rate = 76.7%.
• Residual cohorts: high-residual Q75 = 0.506, low-residual Q25 = 0.417; high-residual errors cover 32.6% of total errors.
• Pattern inventory: 779 candidate latent patterns discovered from SAE diffing.
• Set-level metrics: coverage = 1.000, concentration = 0.767, AUC = 0.498, F1 = 0.575, distinctness = 0.400 (avg Jaccard = 0.600).
• Interpretation: discovery breadth is strong, but ranking power and deduplication remain the primary improvement targets.

Report Graph Highlights

The following summaries are taken from the latest poster report and included here to make key quantitative evidence readable directly from this page.

Error Rate by MMLU Subject (Top 10)

Plot 2 - SAE Latent Delta Heatmap (High - Low Residual)

Each cell shows mean SAE activation on high-residual errors minus low-residual errors for a subject-latent pair. Amber/orange cells indicate the latent activates more on surprising failures, while blue indicates suppression.

Plot 3 - Set-Level Metrics and Interpretation

Limitations

• AUC remains near random (0.498), indicating latent membership alone is still weak for ranking error likelihood.
• Pattern overlap is high (avg Jaccard 0.600), so discovered patterns are not yet sufficiently distinct.
• Prediction-generation and representation-analysis can be misaligned across model setups, which may blur causal interpretation.

Future Work

• Deduplicate patterns with clustering and overlap-aware pruning to improve distinctness.
• Improve predictive ranking quality with richer features and stronger set-level selection criteria.
• Run broader multi-seed consistency analysis (for example overlap@k) to validate stability across runs.

References

• Teaching People LLM's Errors and Getting It Right (Stringham et al., 2025).
• Sparse Autoencoders Find Highly Interpretable Features in Language Models (Cunningham et al., 2023).
• Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit (Jiang et al., 2025).