NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2022-12arXiv 2212.09849Dataless Knowledge Fusion by Merging Weights of Language Models
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro et al. · Language models & NLP
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. Oftentimes fine-tuned models are readily available but their training data is not, due to data privacy or intellectual property concerns.
- 2022-10arXiv 2210.07229Mass-Editing Memory in a Transformer
Kevin Meng, Arnab Sen Sharma, Alex Andonian et al. · Language models & NLP
Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Consequence consistent updatesStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weights - 2022-04arXiv 2204.14211TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Joel Jang, Seonghyeon Ye, Changho Lee et al. · Language models & NLP
Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment.
- 2021-10arXiv 2110.03215Towards Continual Knowledge Learning of Language Models
Joel Jang, Seonghyeon Ye, Sohee Yang et al. · Language models & NLP
Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue.
- 2021-06arXiv 2106.14806Laplace Redux -- Effortless Bayesian Deep Learning
Erik Daxberger, Agustinus Kristiadi, Alexander Immer et al. · Machine learning
Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection.
- 2021-06arXiv 2106.08769Knowledge-Adaptation Priors
Mohammad Emtiyaz Khan, Siddharth Swaroop · Machine learning
Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch.
- 2021-06arXiv 2106.08085Natural continual learning: success is a journey, not (just) a destination
Ta-Chu Kao, Kristopher T. Jensen, Gido M. van de Ven et al. · Machine learning
Biological agents are known to learn many different tasks over the course of their lives, and to be able to revisit previous tasks and behaviors with little to no loss in performance.
- 2021-01arXiv 2101.07592Synaptic metaplasticity in binarized neural networks
Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin et al. · Computational and Systems Neuroscience (Cosyne) 2021 · Neuroscience & neural computation
Unlike the brain, artificial neural networks, including state-of-the-art deep neural networks for computer vision, are subject to "catastrophic forgetting": they rapidly forget the previous task when trained on a new one.
- 2020-06arXiv 2006.06357Unifying Regularisation Methods for Continual Learning
Frederik Benzing · Machine learning
Continual Learning addresses the challenge of learning a number of different tasks sequentially. The goal of maintaining knowledge of earlier tasks without re-accessing them starkly conflicts with standard SGD training for artificial neural networks.
- 2019-11arXiv 1911.04933Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Aditya Golatkar, Alessandro Achille, Stefano Soatto · Machine learning
We explore the problem of selectively forgetting a particular subset of the data used for training a deep neural network. While the effects of the data to be forgotten can be hidden from the output of the network, insights may still be gleaned by probing deep into its weights.
- 2019-05arXiv 1905.12558Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Frederik Kunstner, Lukas Balles, Philipp Hennig · Machine learning
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
- 2019-05arXiv 1905.11614Uncertainty-based Continual Learning with Adaptive Regularization
Hongjoon Ahn, Sungmin Cha, Donggyu Lee et al. · Machine learning
We introduce a new neural network-based continual learning algorithm, dubbed as Uncertainty-regularized Continual Learning (UCL), which builds on traditional Bayesian online learning framework with variational inference.
- 2019-04arXiv 1904.07734Three scenarios for continual learning
Gido M. van de Ven, Andreas S. Tolias · Machine learning
Standard artificial neural networks suffer from the well-known issue of catastrophic forgetting, making continual or lifelong learning difficult for machine learning.
- 2018-05arXiv 1805.07810Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
Hippolyt Ritter, Aleksandar Botev, David Barber · Machine learning
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks.
- 2018-05arXiv 1805.06370Progress & Compress: A scalable framework for continual learning
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki et al. · Machine learning
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceEWC stiffness freezes stale facts, and no per-fact Fisher ledger can release itStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learning - 2018-02arXiv 1802.07239Continual Reinforcement Learning with Complex Synapses
Christos Kaplanis, Murray Shanahan, Claudia Clopath · AI & agents
Unlike humans, who are capable of continual learning over their lifetimes, artificial neural networks have long been known to suffer from a phenomenon known as catastrophic forgetting, whereby new learning can lead to abrupt erasure of previously acquired knowledge.
- 2018-02arXiv 1802.07329Bayesian Incremental Learning for Deep Neural Networks
Max Kochurov, Timur Garipov, Dmitry Podoprikhin et al. · Machine learning
In industrial machine learning pipelines, data often arrive in parts. Particularly in the case of deep neural networks, it may be too expensive to train the model from scratch each time, so one would rather use a previously learned model and the new data to improve performance.
- 2018-02arXiv 1802.02950Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
Xialei Liu, Marc Masana, Luis Herranz et al. · Vision
In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters.
- 2018-01arXiv 1801.10112Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan et al. · Vision
Incremental learning (IL) has received a lot of attention recently, however, the literature lacks a precise problem definition, proper evaluation settings, and metrics tailored specifically for the IL problem.
- 2017-12arXiv 1712.03847On Quadratic Penalties in Elastic Weight Consolidation
Ferenc Huszár · Machine learning
Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks.
- 2017-10arXiv 1710.10628Variational Continual Learning
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui et al. · Machine learning
This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks.
- 2017-03arXiv 1703.08475Overcoming Catastrophic Forgetting by Incremental Moment Matching
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun et al. · Machine learning
Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task. incremental moment matching (IMM), to resolve this problem.
- 2017-03arXiv 1703.01357A Computational Model of Systems Memory Consolidation and Reconsolidation
Peter Helfer, Thomas R. Shultz · Hippocampus, 30, (2020) 659-677 · Neuroscience & neural computation
In the mammalian brain, newly acquired memories depend on the hippocampus for maintenance and recall, but over time the neocortex takes over these functions, rendering memories hippocampus-independent.
- 2016-12arXiv 1612.00796Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz et al. · Machine learning
The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceEWC stiffness freezes stale facts, and no per-fact Fisher ledger can release itStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learning - 2014-12arXiv 1412.1193New insights and perspectives on the natural gradient method
James Martens · Machine learning
Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
