NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2024-01arXiv 2401.02415LLaMA Pro: Progressive LLaMA with Block Expansion
Chengyue Wu, Yukang Gan, Yixiao Ge et al. · Language models & NLP
, from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting.
- 2024-01arXiv 2401.02509Memory, Consciousness and Large Language Model
Jitang Li, Jinzheng Li · Neuroscience & neural computation
With the development in cognitive science and Large Language Models (LLMs), increasing connections have come to light between these two distinct fields. Building upon these connections, we propose a conjecture suggesting the existence of a duality between LLMs and Tulving's theory of memory.
- 2023-11arXiv 2311.08011Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models
Shiwen Ni, Dingwei Chen, Chengming Li et al. · Language models & NLP
Recent advancements in Large Language Models (LLMs) have showcased their remarkable capabilities in text understanding and generation. However, even stronger LLMs are susceptible to acquiring erroneous or obsolete information from the training corpus.
- 2023-11arXiv 2311.08106Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models
Yujin Kim, Jaehong Yoon, Seonghyeon Ye et al. · Language models & NLP
The dynamic nature of knowledge in an ever-changing world presents challenges for language models trained on static data; the model in the real world often requires not only acquiring new knowledge but also overwriting outdated information into updated ones.
- 2023-11arXiv 2311.03099Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
Le Yu, Bowen Yu, Haiyang Yu et al. · Language models & NLP
In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs.
- 2023-10arXiv 2310.14152Orthogonal Subspace Learning for Language Model Continual Learning
Xiao Wang, Tianze Chen, Qiming Ge et al. · Language models & NLP
Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Continual learning order and companion dataStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Genesys program prior artStudy: Hardware guardian core for continual learningStudy: Merging and splitting MoE expertsStudy: Orchestrated specialists and bridge synthesisStudy: Seven continual learning methods tested - 2023-10arXiv 2310.06762TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
Xiao Wang, Yuansen Zhang, Tianze Chen et al. · Language models & NLP
Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. However, the continual learning aspect of these aligned LLMs has been largely overlooked.
- 2023-07arXiv 2307.01163Improving Language Plasticity via Pretraining with Active Forgetting
Yihong Chen, Kelly Marchisio, Roberta Raileanu et al. · Language models & NLP
Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible.
- 2023-06arXiv 2306.13812Maintaining Plasticity in Deep Continual Learning
Shibhansh Dohare, J. Fernando Hernandez-Garcia, Parash Rahman et al. · Machine learning
Modern deep-learning systems are specialized to problem settings in which training occurs once and then never again, as opposed to continual-learning settings in which training occurs continually.
- 2023-06arXiv 2306.01708TIES-Merging: Resolving Interference When Merging Models
Prateek Yadav, Derek Tam, Leshem Choshen et al. · Machine learning
, further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency.
Cited inStudy: Decision signatures for conflict detectionStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learningStudy: Hierarchical and T shaped MoEStudy: LLMs modifying open model weightsStudy: Merging and splitting MoE expertsStudy: Weight editing to prevent forgetting - 2023-02arXiv 2302.12902The Dormant Neuron Phenomenon in Deep Reinforcement Learning
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro et al. · Machine learning
In this work we identify the dormant neuron phenomenon in deep reinforcement learning, where an agent's network suffers from an increasing number of inactive neurons, thereby affecting network expressivity.
- 2023-02arXiv 2302.01386Continual Learning with Scaled Gradient Projection
Gobinda Saha, Kaushik Roy · Proceedings of Thirty-Seventh AAAI Conference on Artificial Intelligence (2023) · Machine learning
In neural networks, continual learning results in gradient interference among sequential tasks, leading to catastrophic forgetting of old tasks while learning new ones.
- 2023-01arXiv 2302.00487A Comprehensive Survey of Continual Learning: Theory, Method and Application
Liyuan Wang, Xingxing Zhang, Hang Su et al. · Machine learning
To cope with real-world dynamics, an intelligent system needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime. This ability, known as continual learning, provides a foundation for AI systems to develop themselves adaptively.
- 2022-12arXiv 2212.04089Editing Models with Task Arithmetic
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman et al. · Machine learning
, improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems.
- 2022-11arXiv 2211.11031Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi et al. · Machine learning
Deployed language models decay over time due to shifting inputs, changing user needs, or emergent world-knowledge gaps. When such problems are identified, we want to make targeted edits while avoiding expensive retraining.
- 2022-10arXiv 2210.07229Mass-Editing Memory in a Transformer
Kevin Meng, Arnab Sen Sharma, Alex Andonian et al. · Language models & NLP
Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Consequence consistent updatesStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weights - 2022-06arXiv 2206.06520Memory-Based Model Editing at Scale
Eric Mitchell, Charles Lin, Antoine Bosselut et al. · AI & agents
Even the largest neural networks make errors, and once-correct predictions can become invalid as the world changes. Model editors make local updates to the behavior of base (pre-trained) models to inject updated knowledge or correct undesirable behaviors.
- 2022-05arXiv 2205.09588How catastrophic can catastrophic forgetting be in linear regression?
Itay Evron, Edward Moroshko, Rachel Ward et al. · 35th Annual Conference on Learning Theory (2022) · Machine learning
To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions.
- 2022-04arXiv 2204.14211TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Joel Jang, Seonghyeon Ye, Changho Lee et al. · Language models & NLP
Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment.
- 2022-03arXiv 2203.05482Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre et al. · Machine learning
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.
- 2022-02arXiv 2202.05262Locating and Editing Factual Associations in GPT
Kevin Meng, David Bau, Alex Andonian et al. · Language models & NLP
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations.
- 2022-01arXiv 2201.12604Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
Elahe Arani, Fahad Sarfraz, Bahram Zonooz · Machine learning
Humans excel at continually learning from an ever-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forgetting.
- 2021-10arXiv 2110.11309Fast Model Editing at Scale
Eric Mitchell, Charles Lin, Antoine Bosselut et al. · Machine learning
While large pre-trained models have enabled impressive results on a variety of downstream tasks, the largest existing models still make errors, and even accurate predictions may become outdated over time.
- 2021-10arXiv 2110.03215Towards Continual Knowledge Learning of Language Models
Joel Jang, Seonghyeon Ye, Sohee Yang et al. · Language models & NLP
Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue.
- 2021-09arXiv 2109.01903Robust fine-tuning of zero-shot models
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim et al. · Vision
, without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts.
- 2021-03arXiv 2103.09762Gradient Projection Memory for Continual Learning
Gobinda Saha, Isha Garg, Kaushik Roy · International Conference on Learning Representations (ICLR), 2021 · Machine learning
The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory.
- 2021-03arXiv 2103.07113Training Networks in Null Space of Feature Covariance for Continual Learning
Shipeng Wang, Xiaorong Li, Jian Sun et al. · Machine learning
In the setting of continual learning, a network is trained on a sequence of tasks, and suffers from catastrophic forgetting.
- 2021-01arXiv 2101.07592Synaptic metaplasticity in binarized neural networks
Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin et al. · Computational and Systems Neuroscience (Cosyne) 2021 · Neuroscience & neural computation
Unlike the brain, artificial neural networks, including state-of-the-art deep neural networks for computer vision, are subject to "catastrophic forgetting": they rapidly forget the previous task when trained on a new one.
- 2020-07arXiv 2007.07400Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Vinay V. Ramasesh, Ethan Dyer, Maithra Raghu · Machine learning
A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks.
- 2020-06arXiv 2006.14769Supermasks in Superposition
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu et al. · Machine learning
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance.
- 2020-06arXiv 2006.11722Type-II Superconductivity below 4K in Sn0.4Sb0.6
M. M. Sharma, Kapil Kumar, Lina Sang et al. · J. Alloys and Compounds (2020) · Hardware, photonics & physics
6 single crystal at below 4K. Rietveld refined Powder XRD data confirms the phase purity of as grown crystal, crystallizing in rhombohedral R-3m space group with an elongated (2xc) unit cell in c-direction.
- 2019-10arXiv 1910.08475On Warm-Starting Neural Network Training
Jordan T. Ash, Ryan P. Adams · 2020 Advances in Neural Information Processing Systems · Machine learning
In many real-world deployments of machine learning systems, data arrive piecemeal., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data.
- 2019-05arXiv 1905.12558Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Frederik Kunstner, Lukas Balles, Philipp Hennig · Machine learning
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
- 2019-01arXiv 1901.11356Functional Regularisation for Continual Learning with Gaussian Processes
Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews et al. · Machine learning
We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network.
- 2018-12arXiv 1812.00420Efficient Lifelong Learning with A-GEM
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach et al. · Machine learning
In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task.
- 2018-09arXiv 1810.01256Continual Learning of Context-dependent Processing in Neural Networks
Guanxiong Zeng, Yang Chen, Bo Cui et al. · Machine learning
Deep neural networks (DNNs) are powerful tools in learning sophisticated but fixed mapping rules between inputs and outputs, thereby limiting their application in more complex and dynamic situations in which the mapping rules are not kept the same but changing according to different contexts.
- 2018-05arXiv 1805.06370Progress & Compress: A scalable framework for continual learning
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki et al. · Machine learning
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceEWC stiffness freezes stale facts, and no per-fact Fisher ledger can release itStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learning - 2018-02arXiv 1802.01569Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
Nicolas Y. Masse, Gregory D. Grant, David J. Freedman · Proceedings of the National Academy of Sciences, 115(44), E10467-E10475 · Machine learning
Humans and most animals can learn new tasks without forgetting old ones. However, training artificial neural networks (ANNs) on new tasks typically cause it to forget previously learned tasks.
- 2018-01arXiv 1801.10112Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan et al. · Vision
Incremental learning (IL) has received a lot of attention recently, however, the literature lacks a precise problem definition, proper evaluation settings, and metrics tailored specifically for the IL problem.
- 2018-01arXiv 1801.06519Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
Arun Mallya, Dillon Davis, Svetlana Lazebnik · Vision
This work presents a method for adapting a single, fixed deep neural network to multiple tasks without affecting performance on already learned tasks.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
