NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2026-06arXiv 2606.09899When Attribution Patching Lies: Diagnosis and a Second-Order Correction
Luyang Zhang, Jialu Wang · Machine learning
A central goal of mechanistic interpretability is to identify which internal components causally drive a language model's behavior. Because these importance estimates serve as the evidence for identifying circuits, systematic errors can lead to the misidentification of the underlying mechanisms.
- 2026-05arXiv 2605.12128Metaphor Is Not All Attention Needs
Olga Sorokoletova, Francesco Giarrusso, Giacomo De Luca et al. · Language models & NLP
Large language models are increasingly deployed in safety-critical applications, where their ability to resist harmful instructions is essential.
- 2026-02arXiv 2602.14111Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?
Anton Korznikov, Andrey Galichin, Alexey Dontsov et al. · Machine learning
Sparse Autoencoders (SAEs) have emerged as a promising tool for interpreting neural networks by decomposing their activations into sparse sets of human-interpretable features.
- 2026-02arXiv 2602.02132There Is More to Refusal in Large Language Models than a Single Direction
Faaiz Joad, Majd Hawasly, Sabri Boughorbel et al. · Language models & NLP
Prior work argues that refusal in large language models is mediated by a single direction, enabling steering and abliteration.
- 2026-01arXiv 2601.21183Sycophantic Anchors: Localizing and Quantifying User Agreement in Reasoning Models
Jacek Duszenko · AI & agents
Reasoning models frequently agree with incorrect user suggestions -- a behavior known as sycophancy. However, it is unclear where in the reasoning trace this agreement originates and how strong the commitment is.
- 2026-01arXiv 2601.21577Collaborative Parameter Learning: Mitigating Forgetting via Parameter-Level Gradient Analysis
Mutian Yang, Zisen Zhan, Yutong Chen et al. · Machine learning
Catastrophic forgetting during knowledge injection impairs the ability of large language models to acquire new knowledge without overwriting previously mastered knowledge.
- 2026-01arXiv 2601.18699Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov · Machine learning
Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities.
- 2026-01arXiv 2601.16644Sycophancy Hides Linearly in the Attention Heads
Rifo Genadi, Munachiso Nwadike, Nurdaulet Mukhituly et al. · Language models & NLP
We find that correct-to-incorrect sycophancy signals are most linearly separable within multi-head attention activations.
- 2026-01arXiv 2602.02496The Hypocrisy Gap: Quantifying Divergence Between Internal Belief and Chain-of-Thought Explanation via Sparse Autoencoders
Shikhar Shiromani, Archie Chaudhury, Sri Pranav Kunda · Language models & NLP
Large Language Models (LLMs) frequently exhibit unfaithful behavior, producing a final answer that differs significantly from their internal chain of thought (CoT) reasoning in order to appease the user they are conversing with.
- 2025-11arXiv 2511.13653Weight-sparse transformers have interpretable circuits
Leo Gao, Achyuta Rajaram, Jacob Coxon et al. · Machine learning
Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be zeros, so that each neuron only has a few connections.
- 2025-10arXiv 2510.19116That's Deprecated! Understanding, Detecting, and Steering Knowledge Conflicts in Language Models for Code Generation
Jaesung Bae, Cameron Churchwell, Mitchell Hermon et al. · Language models & NLP
This paper investigates how large language models (LLMs) behave when faced with discrepancies between their parametric knowledge and conflicting information contained in a prompt.
- 2025-09arXiv 2509.21344Linear probes rely on textual evidence: Results from leakage mitigation studies in language models
Gerard Boxo, Aman Neelappa, Shivam Raval · AI & agents
White-box monitors are a popular technique for detecting potentially harmful behaviours in language models. While they perform well in general, their effectiveness in detecting text-ambiguous behaviour is disputed.
- 2025-08arXiv 2508.09363Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders
Charles O'Neill, Mudith Jayasekara, Max Kirkby · Machine learning
Sparse autoencoders (SAEs) decompose large language model (LLM) activations into latent features that reveal mechanistic structure. Conventional SAEs train on broad data distributions, forcing a fixed latent budget to capture only high-frequency, generic patterns.
- 2025-07arXiv 2508.00161Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs
Ziqian Zhong, Aditi Raghunathan · Machine learning
The releases of powerful open-weight large language models (LLMs) are often not accompanied by access to their full training data. Existing interpretability methods, particularly those based on activations, often require or assume distributionally similar data.
- 2025-06arXiv 2506.23423TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs
Felipe Nuti, Tim Franzmeyer, João Henriques · Language models & NLP
Past work has studied the effects of fine-tuning on large language models' (LLMs) overall performance on certain tasks. However, a quantitative and systematic method for analyzing its effect on individual outputs is still lacking.
- 2025-04arXiv 2504.14530Causality for Natural Language Processing
Zhijing Jin · Language models & NLP
Causal reasoning is a cornerstone of human intelligence and a critical capability for artificial systems aiming to achieve advanced understanding and decision-making.
- 2025-04arXiv 2504.13151MIB: A Mechanistic Interpretability Benchmark
Aaron Mueller, Atticus Geiger, Sarah Wiegreffe et al. · Machine learning
How can we know whether new mechanistic interpretability methods achieve real improvements? In pursuit of lasting evaluation standards, we propose MIB, a Mechanistic Interpretability Benchmark, with two tracks spanning four tasks and five models.
- 2025-02arXiv 2502.16681Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan et al. · Machine learning
Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations.
- 2025-02arXiv 2502.03407Detecting Strategic Deception Using Linear Probes
Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim et al. · Machine learning
AI models might use deceptive strategies as part of scheming or misaligned behaviour. Monitoring outputs alone is insufficient, since the AI might produce seemingly benign outputs while their internal reasoning is misaligned.
- 2025-01arXiv 2501.17148AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
Zhengxuan Wu, Aryaman Arora, Atticus Geiger et al. · Language models & NLP
Fine-grained steering of language model outputs is essential for safety and reliability.
- 2024-12arXiv 2412.09565Obfuscated Activations Bypass LLM Latent-Space Defenses
Luke Bailey, Alex Serrano, Abhay Sheshadri et al. · Machine learning
Recent latent-space monitoring techniques have shown promise as defenses against LLM attacks. These defenses act as scanners that seek to detect harmful activations before they lead to undesirable actions.
- 2024-10arXiv 2410.20526Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
Zhengfu He, Wentao Shu, Xuyang Ge et al. · Machine learning
Sparse Autoencoders (SAEs) have emerged as a powerful unsupervised method for extracting sparse representations from language models, yet scalable training remains a significant challenge.
- 2024-10arXiv 2410.17413Scalable Influence and Fact Tracing for Large Language Model Pretraining
Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi et al. · Language models & NLP
Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples, and the application of these methods to large language model (LLM) outputs could significantly advance model transparency and data curation.
- 2024-10arXiv 2410.16090Analysing the Residual Stream of Language Models Under Knowledge Conflicts
Yu Zhao, Xiaotang Du, Giwon Hong et al. · Language models & NLP
Large language models (LLMs) can store a significant amount of factual knowledge in their parameters. However, their parametric knowledge may conflict with the information provided in the context.
- 2024-10arXiv 2410.14670Decomposing The Dark Matter of Sparse Autoencoders
Joshua Engels, Logan Riggs, Max Tegmark · Machine learning
Sparse autoencoders (SAEs) are a promising technique for decomposing language model activations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in "dark matter": unexplained variance in activations.
- 2024-10arXiv 2410.05817Probing Language Models on Their Knowledge Source
Zineddine Tighidet, Andrea Mogini, Jiali Mei et al. · Language models & NLP
Large Language Models (LLMs) often encounter conflicts between their learned, internal (parametric knowledge, PK) and external knowledge provided during inference (contextual knowledge, CK).
- 2024-10arXiv 2410.02707LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
Hadas Orgad, Michael Toker, Zorik Gekhman et al. · Language models & NLP
Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations".
- 2024-08arXiv 2408.14677Can Optimization Trajectories Explain Multi-Task Transfer?
David Mueller, Mark Dredze, Nicholas Andrews · Machine learning
Despite the widespread adoption of multi-task training in deep learning, little is understood about how multi-task learning (MTL) affects generalization.
- 2024-08arXiv 2408.12076ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM
Zhaochen Su, Jun Zhang, Xiaoye Qu et al. · Language models & NLP
Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts, a major source of hallucinations, has rarely been studied.
- 2024-07arXiv 2407.12404Analyzing the Generalization and Reliability of Steering Vectors
Daniel Tan, David Chanin, Aengus Lynch et al. · Machine learning
Steering vectors (SVs) have been proposed as an effective approach to adjust language model behaviour at inference time by intervening on intermediate model activations.
- 2024-06arXiv 2406.15927Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
Jannik Kossen, Jiatong Han, Muhammed Razzak et al. · Language models & NLP
We propose semantic entropy probes (SEPs), a cheap and reliable method for uncertainty quantification in Large Language Models (LLMs). Hallucinations, which are plausible-sounding but factually incorrect and arbitrary model generations, present a major challenge to the practical adoption of LLMs.
Cited inStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errorsStudy: Decision signatures for conflict detectionStudy: Logical coherence of LLM world modelsStudy: Model self knowledge and confident errorsStudy: Stopping confident wrong expert votes - 2024-06arXiv 2406.11717Refusal in Language Models Is Mediated by a Single Direction
Andy Arditi, Oscar Obeso, Aaquib Syed et al. · Machine learning
Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones.
- 2024-06arXiv 2406.00799Get my drift? Catching LLM Task Drift with Activation Deltas
Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin et al. · Security
LLMs are commonly used in retrieval-augmented applications to execute user instructions based on data from external sources. For example, modern search engines use LLMs to answer queries based on relevant search results; email plugins summarize emails by processing their content through an LLM.
- 2024-05arXiv 2405.13954What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Sang Keun Choe, Hwijeen Ahn, Juhan Bae et al. · Machine learning
Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited.
- 2024-05arXiv 2405.02421What does the Knowledge Neuron Thesis Have to do with Knowledge?
Jingcheng Niu, Andrew Liu, Zining Zhu et al. · Language models & NLP
We reassess the Knowledge Neuron (KN) Thesis: an interpretation of the mechanism underlying the ability of large language models to recall facts from a training corpus.
- 2024-03arXiv 2403.17806Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms
Michael Hanna, Sandro Pezzelle, Yonatan Belinkov · Machine learning
Many recent language model (LM) interpretability studies have adopted the circuits framework, which aims to find the minimal computational subgraph, or circuit, that explains LM behavior on a given task.
- 2024-02arXiv 2402.13494GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis
Yueqi Xie, Minghong Fang, Renjie Pi et al. · Language models & NLP
Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs.
- 2024-02arXiv 2402.11655Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals
Francesco Ortu, Zhijing Jin, Diego Doimo et al. · Language models & NLP
Interpretability research aims to bridge the gap between empirical success and our scientific understanding of the inner workings of large language models (LLMs).
- 2024-02arXiv 2402.04333LESS: Selecting Influential Data for Targeted Instruction Tuning
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan et al. · Language models & NLP
Instruction tuning has unlocked powerful capabilities in large language models (LLMs), effectively using combined datasets to develop generalpurpose chatbots., reasoning).
- 2024-02arXiv 2402.01865What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement
Xisen Jin, Xiang Ren · Machine learning
Language models deployed in the wild make errors. However, simply updating the model with the corrected error instances causes catastrophic forgetting -- the updated model makes errors on instances learned during the instruction tuning or upstream training phase.
Cited inA gradient kernel flags contradictions but does not predict forgetting, and steering rehearsal by it backfiresCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Continual learning order and companion dataStudy: Decision signatures for conflict detectionStudy: Logical coherence of LLM world modelsStudy: Seven continual learning methods tested
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
