NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2026-09arXiv 2609.23916Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Fırat Öncel, Salman Hussain Ali, Mirco Ravanelli et al. · Language models & NLP
Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Continued pretraining (CPT) is the natural remedy, but it is typically evaluated through a continual learning lens that assumes disjoint data streams.
- 2026-09arXiv 2609.03900Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating
Heejin Choi · Machine learning
Continual knowledge-updating methods are often declared superior from one final checkpoint and one conventional adapter rank. We show that this can be insufficient to identify the better operating point.
- 2026-08arXiv 2608.01437Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning
Huiyu Yi, Yongqi Xu, Bogang Zhang et al. · AI & agents
Multimodal Continual Instruction Tuning (MCIT) enables multimodal large language models to acquire new tasks sequentially while retaining previously learned capabilities.
- 2026-06arXiv 2606.24993The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order
John Sweeney · Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:117806-117825, 2026 · Machine learning
possible curricula. We show that the local order effect is governed by a computable geometric quantity, the Lie-bracket commutator of gradient update fields, yielding a pairwise score for whether A->B or B->A is better for a target domain.
- 2026-06arXiv 2606.23063Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
Chuangxin Zhao, Canran Xiao, Siyuan Ma et al. · Vision
Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions, yet continual fine-tuning often causes severe forgetting of previously acquired multimodal skills.
- 2026-05arXiv 2605.31025TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
Xiaosong Han, Ke Chen, Xindi Dai et al. · Language models & NLP
In real-world deployment, LLMs are often adapted continually across tasks to keep LLMs up-to-date in production, where new fine-tuning should preserve previously learned skills.
- 2026-05arXiv 2605.29495On-Policy Replay for Continual Supervised Fine-Tuning
Yan Chen, Taojie Zhu, Meng Zhang et al. · Machine learning
Continual supervised fine-tuning (SFT) is the de facto recipe for adapting large language models (LLMs) to a stream of downstream tasks, but it suffers from catastrophic forgetting of earlier capabilities.
- 2026-05arXiv 2605.11836More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
Xin Ma, Wei Chen, Qi Liu et al. · Machine learning
Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities, yet it remains plagued by catastrophic forgetting and model collapse.
- 2026-04arXiv 2604.10079Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models
Chao Xue, Yao Wang, Mengqiao Liu et al. · Language models & NLP
Supervised Fine-Tuning (SFT) is the standard approach for adapting large language models (LLMs) to downstream tasks. However, we observe a persistent failure mode: even after convergence, models often fail to correctly reproduce a subset of their own supervised training data.
- 2026-03arXiv 2603.09892MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning
Yiyang Lu, Yu He, Jianlong Chen et al. · Machine learning
Continual fine-tuning of large language models (LLMs) is becoming increasingly crucial as these models are deployed in dynamic environments where tasks and data distributions evolve over time.
- 2026-01arXiv 2602.02568Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation
Protik Nag, Krishnan Raghavan, Vignesh Narayanan · Machine learning
We introduce , a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering.
- 2026-01arXiv 2602.02543Norm Anchors Make Model Edits Last
Mingda Liu, Zhenghan Zhu, Ze'an Miao et al. · Machine learning
Sequential Locate-and-Edit (L&E) model editing can fail abruptly after many edits.
- 2026-01arXiv 2601.21577Collaborative Parameter Learning: Mitigating Forgetting via Parameter-Level Gradient Analysis
Mutian Yang, Zisen Zhan, Yutong Chen et al. · Machine learning
Catastrophic forgetting during knowledge injection impairs the ability of large language models to acquire new knowledge without overwriting previously mastered knowledge.
- 2026-01arXiv 2601.21698Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
Mohamed Elgaar, Hadi Amiri · Machine learning
Curriculum learning changes the order of pretraining data, but it remains unclear how ordering changes the learning dynamics.
- 2026-01arXiv 2601.18699Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov · Machine learning
Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities.
- 2025-12arXiv 2512.16871Sequencing to Mitigate Catastrophic Forgetting in Continual Learning
Hesham G. Moussa, Aroosa Hameed, Arashmid Akhavain · Machine learning
To cope with real-world dynamics, an intelligent system needs to incrementally acquire, update, and exploit knowledge throughout its lifetime. This ability, known as Continual learning, provides a foundation for AI systems to develop themselves adaptively.
- 2025-10arXiv 2510.26336From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learning
Nishit Neema, Srinjoy Mukherjee, Sapan Shah et al. · Language models & NLP
Large Language Models (LLMs) excel at general tasks but underperform in specialized domains like economics and psychology, which require deep, principled understanding.
- 2025-10arXiv 2510.19941Are Greedy Task Orderings Better Than Random in Continual Linear Regression?
Matan Tsipory, Ran Levinstein, Itay Evron et al. · Machine learning
We analyze task orderings in continual learning for linear regression, assuming joint realizability of training data. We focus on orderings that greedily maximize dissimilarity between consecutive tasks, a concept briefly explored in prior work but still surrounded by open questions.
- 2025-07arXiv 2508.08275MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis
Haiyun Guo, Zhiyan Hou, Yandu Sun et al. · Language models & NLP
Continual instruction tuning(CIT) during the post-training phase is crucial for adapting multimodal large language models (MLLMs) to evolving real-world demands.
- 2025-06arXiv 2506.10943Self-Adapting Language Models
Adam Zweiger, Jyothish Pari, Han Guo et al. · Machine learning
Large language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce Self-Adapting LLMs (SEAL), a framework that enables LLMs to self-adapt by generating their own finetuning data and update directives.
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsA model rewrote another model's weights. Read what it took away.Study: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weightsStudy: Logical coherence of LLM world modelsStudy: Seven continual learning methods testedStudy: Weight editing to prevent forgetting - 2025-05arXiv 2505.11876EAMET: Robust Massive Model Editing via Embedding Alignment Optimization
Yanbo Dai, Zhenlan Ji, Zongjie Li et al. · Language models & NLP
Model editing techniques are essential for efficiently updating knowledge in large language models (LLMs). However, the effectiveness of existing approaches degrades in massive editing scenarios, particularly when evaluated with practical metrics.
- 2025-04arXiv 2504.09522How new data permeates LLM knowledge and how to dilute it
Chen Sun, Renat Aksitov, Andrey Zhmoginov et al. · Language models & NLP
Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood.
- 2025-04arXiv 2504.02107TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
Jeffrey Li, Mohammadreza Armandpour, Iman Mirzadeh et al. · Machine learning
Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Continual learning order and companion dataStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Hardware guardian core for continual learning - 2025-03arXiv 2503.05683WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Lukas Thede, Karsten Roth, Matthias Bethge et al. · Language models & NLP
Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weightsStudy: Logical coherence of LLM world modelsStudy: Orchestrated specialists and bridge synthesisStudy: Weight editing to prevent forgetting - 2025-02arXiv 2502.03350Optimal Task Order for Continual Learning of Multiple Tasks
Ziyan Li, Naoki Hiratani · Machine learning
Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose a strategy for optimizing it.
- 2025-01arXiv 2501.12774LLMs as Repositories of Factual Knowledge: Limitations and Solutions
Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi · Language models & NLP
). Such unstructured knowledge is subject to change due to updates through time from past to present. Equally important are the inconsistencies and inaccuracies occurring in different information sources.
- 2024-11arXiv 2411.07175Continual Memorization of Factoids in Language Models
Howard Chen, Jiayi Geng, Adithya Bhaskar et al. · Transactions on Machine Learning Research, 2026 · Language models & NLP
As new knowledge rapidly accumulates, language models (LMs) with pretrained knowledge quickly become obsolete. A common approach to updating LMs is fine-tuning them directly on new knowledge.
- 2024-09arXiv 2409.07431Synthetic continued pretraining
Zitong Yang, Neil Band, Shuangping Li et al. · Machine learning
Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it.
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsStudy: Continual learning order and companion dataStudy: Generational competitive continual learningStudy: Logical coherence of LLM world modelsStudy: Orchestrated specialists and bridge synthesis - 2024-07arXiv 2407.09435MUSCLE: A Model Update Strategy for Compatible LLM Evolution
Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli et al. · AI & agents
Large Language Models (LLMs) are regularly updated to enhance performance, typically through changes in data or architecture. Within the update process, developers often prioritize improving overall performance metrics, paying less attention to maintaining compatibility with earlier model versions.
- 2024-06arXiv 2406.14026Demystifying Language Model Forgetting with Low-rank Example Associations
Xisen Jin, Xiang Ren · Machine learning
Large language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks.
- 2024-06arXiv 2406.11813How Do Large Language Models Acquire Factual Knowledge During Pretraining?
Hoyeon Chang, Jinho Park, Seonghyeon Ye et al. · Language models & NLP
Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining.
- 2024-05arXiv 2405.05904Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Zorik Gekhman, Gal Yona, Roee Aharoni et al. · Language models & NLP
When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training.
Cited inStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errorsTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learningStudy: Graded succession by earned authorityStudy: Logical coherence of LLM world modelsStudy: Model self knowledge and confident errorsStudy: Orchestrated specialists and bridge synthesisStudy: Stopping confident wrong expert votes - 2024-03arXiv 2403.08350CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
Cheng Chen, Junchen Zhu, Xu Luo et al. · Vision
Instruction tuning represents a prevalent strategy employed by Multimodal Large Language Models (MLLMs) to align with human instructions and adapt to new tasks. Nevertheless, MLLMs encounter the challenge of adapting to users' evolving knowledge and demands.
- 2024-03arXiv 2403.07175Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing
Akshat Gupta, Sidharth Baskaran, Gopala Anumanchipalli · Language models & NLP
Recent work using Rank-One Model Editing (ROME), a popular model editing method, has shown that there are certain facts that the algorithm is unable to edit without breaking the model.
- 2024-03arXiv 2403.01244Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal
Jianheng Huang, Leyang Cui, Ante Wang et al. · Language models & NLP
Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications.
- 2024-02arXiv 2402.17400Investigating Continual Pretraining in Large Language Models: Insights and Implications
Çağatay Yıldız, Nishaanth Kanna Ravichandran, Nitin Sharma et al. · Language models & NLP
Continual learning (CL) in large language models (LLMs) is an evolving domain that focuses on developing efficient and sustainable training strategies to adapt models to emerging knowledge and achieve robustness in dynamic environments.
- 2024-02arXiv 2402.01865What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement
Xisen Jin, Xiang Ren · Machine learning
Language models deployed in the wild make errors. However, simply updating the model with the corrected error instances causes catastrophic forgetting -- the updated model makes errors on instances learned during the instruction tuning or upstream training phase.
Cited inA gradient kernel flags contradictions but does not predict forgetting, and steering rehearsal by it backfiresCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Continual learning order and companion dataStudy: Decision signatures for conflict detectionStudy: Logical coherence of LLM world modelsStudy: Seven continual learning methods tested - 2023-12arXiv 2312.05934Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
Oded Ovadia, Menachem Brief, Moshik Mishaeli et al. · AI & agents
Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains.
- 2023-10arXiv 2310.16226TiC-CLIP: Continual Training of CLIP Models
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari et al. · Vision
Keeping large foundation models up to date on latest data is inherently expensive. To avoid the prohibitive costs of constantly retraining, it is imperative to continually train these models.
- 2023-10arXiv 2310.14152Orthogonal Subspace Learning for Language Model Continual Learning
Xiao Wang, Tianze Chen, Qiming Ge et al. · Language models & NLP
Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Continual learning order and companion dataStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Genesys program prior artStudy: Hardware guardian core for continual learningStudy: Merging and splitting MoE expertsStudy: Orchestrated specialists and bridge synthesisStudy: Seven continual learning methods tested
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
