NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2023-12arXiv 2312.10029Challenges with unsupervised LLM knowledge discovery
Sebastian Farquhar, Vikrant Varma, Zachary Kenton et al. · Machine learning
We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent.
- 2023-11arXiv 2311.03099Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
Le Yu, Bowen Yu, Haiyang Yu et al. · Language models & NLP
In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs.
- 2023-10arXiv 2310.15910Characterizing Mechanisms for Factual Recall in Language Models
Qinan Yu, Jack Merullo, Ellie Pavlick · Language models & NLP
Language Models (LMs) often must integrate facts they memorized in pretraining with new information that appears in a given context. These two sources can disagree, causing competition within the model, and it is unclear how an LM will resolve the conflict.
- 2023-10arXiv 2310.10348Attribution Patching Outperforms Automated Circuit Discovery
Aaquib Syed, Can Rager, Arthur Conmy · Machine learning
Automated interpretability research has recently attracted attention as a potential research direction that could scale explanations of neural network behavior to large models.
- 2023-10arXiv 2310.00902DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models
Yongchan Kwon, Eric Wu, Kevin Wu et al. · Machine learning
Quantifying the impact of training data points is crucial for understanding the outputs of machine learning models and for improving the transparency of the AI pipeline.
- 2023-10arXiv 2310.01405Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou, Long Phan, Sarah Chen et al. · Machine learning
In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience.
- 2023-08arXiv 2308.03296Studying Large Language Model Generalization with Influence Functions
Roger Grosse, Juhan Bae, Cem Anil et al. · Machine learning
When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior?
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Hardware guardian core for continual learningStudy: Logical coherence of LLM world models - 2023-07arXiv 2307.12976Evaluating the Ripple Effects of Knowledge Editing in Language Models
Roi Cohen, Eden Biran, Ori Yoran et al. · Language models & NLP
Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations.
Cited inA gradient kernel flags contradictions but does not predict forgetting, and steering rehearsal by it backfiresCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceIterated self-closure settles fast, but only evidence that overrides self-belief makes the settled state trueTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Graded succession by earned authorityStudy: Hardware guardian core for continual learningStudy: Logical coherence of LLM world modelsStudy: Orchestrated specialists and bridge synthesis - 2023-07arXiv 2307.09476Overthinking the Truth: Understanding how Language Models Process False Demonstrations
Danny Halawi, Jean-Stanislas Denain, Jacob Steinhardt · Machine learning
Modern language models can imitate complex patterns through few-shot learning, enabling them to complete challenging tasks without fine-tuning. However, imitation can also lead models to reproduce inaccuracies or harmful content if present in the context.
- 2023-06arXiv 2306.01708TIES-Merging: Resolving Interference When Merging Models
Prateek Yadav, Derek Tam, Leshem Choshen et al. · Machine learning
, further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency.
Cited inStudy: Decision signatures for conflict detectionStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learningStudy: Hierarchical and T shaped MoEStudy: LLMs modifying open model weightsStudy: Merging and splitting MoE expertsStudy: Weight editing to prevent forgetting - 2023-05arXiv 2305.12827Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models
Guillermo Ortiz-Jimenez, Alessandro Favero, Pascal Frossard · Advances in Neural Information Processing Systems 36 (NeurIPS 2023) · Machine learning
Task arithmetic has recently emerged as a cost-effective and scalable approach to edit pre-trained models directly in weight space: By adding the fine-tuned weights of different tasks, the model's performance can be improved on these tasks, while negating them leads to task forgetting.
- 2023-05arXiv 2305.13300Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
Jian Xie, Kai Zhang, Jiangjie Chen et al. · Language models & NLP
By providing external information to large language models (LLMs), tool augmentation (including retrieval augmentation) has emerged as a promising solution for addressing the limitations of LLMs' static parametric memory.
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsSuccessors should earn authority, not win contests: graded succession in continual learningStudy: Decision signatures for conflict detectionStudy: Graded succession by earned authorityStudy: Logical coherence of LLM world modelsStudy: Memory world models and source trustStudy: Model self knowledge and confident errors - 2023-04arXiv 2304.14997Towards Automated Circuit Discovery for Mechanistic Interpretability
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch et al. · Machine learning
Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed.
- 2023-03arXiv 2303.14186TRAK: Attributing Model Behavior at Scale
Sung Min Park, Kristian Georgiev, Andrew Ilyas et al. · Machine learning
The goal of data attribution is to trace model predictions back to training data. Despite a long line of work towards this goal, existing approaches to data attribution tend to force users to choose between computational tractability and efficacy.
- 2023-03arXiv 2303.08112Eliciting Latent Predictions from Transformers with the Tuned Lens
Nora Belrose, Igor Ostrovsky, Lev McKinney et al. · Machine learning
We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer.
- 2023-01arXiv 2301.04213Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
Peter Hase, Mohit Bansal, Been Kim et al. · Machine learning
Language models learn a great quantity of factual information during pretraining, and recent work localizes this information to specific model weights like mid-layer MLP weights.
- 2022-12arXiv 2212.04089Editing Models with Task Arithmetic
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman et al. · Machine learning
, improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems.
- 2022-10arXiv 2210.07229Mass-Editing Memory in a Transformer
Kevin Meng, Arnab Sen Sharma, Alex Andonian et al. · Language models & NLP
Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Consequence consistent updatesStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weights - 2022-09arXiv 2209.11379Do Current Multi-Task Optimization Methods in Deep Learning Even Help?
Derrick Xin, Behrooz Ghorbani, Ankush Garg et al. · Machine learning
Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply optimizing a weighted average of the task losses.
- 2022-09arXiv 2209.05364If Influence Functions are the Answer, Then What is the Question?
Juhan Bae, Nathan Ng, Alston Lo et al. · Machine learning
Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters. While influence estimates align well with leave-one-out retraining for linear models, recent works have shown this alignment is often poor in neural networks.
- 2022-02arXiv 2202.05262Locating and Editing Factual Associations in GPT
Kevin Meng, David Bau, Alex Andonian et al. · Language models & NLP
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations.
- 2022-01arXiv 2201.04122In Defense of the Unitary Scalarization for Deep Multi-Task Learning
Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov et al. · Machine learning
Recent multi-task learning research argues against unitary scalarization, where training simply minimizes the sum of the task losses. Several ad-hoc multi-task optimization algorithms have instead been proposed, inspired by various hypotheses about what makes multi-task settings difficult.
- 2021-11arXiv 2111.09832Merging Models with Fisher-Weighted Averaging
Michael Matena, Colin Raffel · Machine learning
Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities.
- 2021-10arXiv 2110.14048Conflict-Averse Gradient Descent for Multi-task Learning
Bo Liu, Xingchao Liu, Xiaojie Jin et al. · Machine learning
The goal of multi-task learning is to enable more efficient learning than single task learning by sharing model structures for a diverse set of tasks. A standard multi-task learning objective is to minimize the average loss across all tasks.
- 2021-10arXiv 2110.11309Fast Model Editing at Scale
Eric Mitchell, Charles Lin, Antoine Bosselut et al. · Machine learning
While large pre-trained models have enabled impressive results on a variety of downstream tasks, the largest existing models still make errors, and even accurate predictions may become outdated over time.
- 2021-04arXiv 2104.08696Knowledge Neurons in Pretrained Transformers
Damai Dai, Li Dong, Yaru Hao et al. · Language models & NLP
Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.
- 2020-10arXiv 2010.06808Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
Zhao Chen, Jiquan Ngiam, Yanping Huang et al. · Machine learning
The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights.
- 2020-10arXiv 2010.05874Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Zirui Wang, Yulia Tsvetkov, Orhan Firat et al. · Language models & NLP
Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization.
- 2020-06arXiv 2006.14651Influence Functions in Deep Learning Are Fragile
Samyadeep Basu, Philip Pope, Soheil Feizi · Machine learning
Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation.
- 2020-02arXiv 2002.08484Estimating Training Data Influence by Tracing Gradient Descent
Garima Pruthi, Frederick Liu, Mukund Sundararajan et al. · Machine learning
We introduce a method called TracIn that computes the influence of a training example on a prediction made by the model. The idea is to trace how the loss on the test point changes during the training process whenever the training example of interest was utilized.
- 2020-01arXiv 2001.06782Gradient Surgery for Multi-Task Learning
Tianhe Yu, Saurabh Kumar, Abhishek Gupta et al. · Machine learning
While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge.
- 2019-05arXiv 1905.12558Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Frederik Kunstner, Lukas Balles, Philipp Hennig · Machine learning
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
- 2017-11arXiv 1711.09601Memory Aware Synapses: Learning what (not) to forget
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny et al. · Vision
Humans can learn in a continuous manner. Old rarely utilized knowledge can be overwritten by new incoming information while important, frequently used knowledge is prevented from being erased.
- 2017-03arXiv 1703.04987Equilibrated flux a posteriori error estimates in -norms for high-order discretizations of parabolic problems
Alexandre Ern, Iain Smears, Martin Vohralik · IMA Journal of Numerical Analysis, 2018 · Statistics & theory
We consider the a posteriori error analysis of fully discrete approximations of parabolic problems based on conforming -finite element methods in space and an arbitrary order discontinuous Galerkin method in time.
- 2017-03arXiv 1703.04730Understanding Black-box Predictions via Influence Functions
Pang Wei Koh, Percy Liang · Machine learning
How can we explain the predictions of a black-box model?
- 2016-12arXiv 1612.00796Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz et al. · Machine learning
The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceEWC stiffness freezes stale facts, and no per-fact Fisher ledger can release itStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learning - 2016-02arXiv 1602.04938"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin · Machine learning
Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
