NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2026-11arXiv 2611.12240Prune, Don't Merge: Expert Compression of Mixture-of-Experts LLMs on Multilingual Production Tasks
Tomas Rasymas, Povilas Kvedaras · Language models & NLP
Mixture-of-Experts language models are cheap to run per token but expensive to hold in GPU memory, which drives the cost of serving them in production. One-shot expert compression shrinks them after a single calibration pass.
- 2026-11arXiv 2611.04931Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning
Kun Gu, De Cheng, Zhipeng Xu et al. · Machine learning
Continual learning (CL) aims to adapt models to new tasks while preserving previously learned knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL due to its efficiency and scalability.
- 2026-10arXiv 2611.00544Skill++: Towards Lifelong Agents via Routing-Aware Skill Refinement
Hanrong Zhang, Minghao Liang, Jiawei Wu et al. · AI & agents
Large language model (LLM) agents extend their capabilities with skills, each with a name, a description, and a body. Under progressive disclosure, an LLM router selects skills by name and description alone and loads only the selected bodies.
- 2026-10arXiv 2610.27123Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin et al. · Machine learning
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters.
- 2026-08arXiv 2608.04454Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
Hongyu Zhang, Cheng Yan, Xiang Xia et al. · Vision
Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool.
- 2026-07arXiv 2607.28308Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Huiyuan Tian, Bonan Xu, Shijian Li · Machine learning
Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions.
- 2026-05arXiv 2605.29350ConMoE: MoE Compression as Expert Selection and Functional Assignment
Yilun Yao, Jiaming Pan, Elsie Dai et al. · AI & agents
Sparse Mixture-of-Experts (MoE) models scale model capacity through sparse activation, but storing their full parameters remains costly in GPU memory.
- 2026-05arXiv 2605.28207Pruning and Distilling Mixture-of-Experts into Dense Language Models
Junhyuck Kim, Jihun Yun, Haechan Kim et al. · Language models & NLP
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment.
- 2026-05arXiv 2605.20803Tunable MAGMAX: Preference-Aware Model Merging for Continual Learning
Kei Hiroshima, Kento Uchida, Shinichi Shirakawa · Machine learning
Continual learning (CL) aims to train models sequentially on multiple tasks while mitigating catastrophic forgetting of previously learned knowledge.
- 2026-05arXiv 2605.20247CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
Yang Liu, Toan Nguyen, Flora D. Salim · Machine learning
Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs).
- 2026-05arXiv 2605.13997HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
Tao Zhong, Dongzhe Zheng, Christine Allen-Blanchette · Machine learning
Sparse Mixture-of-Experts (MoE) layers route tokens through a handful of experts, and learning-free compression of these layers reduces inference cost without retraining.
- 2026-05arXiv 2605.07494DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
Mengxin Qin, Xiang Zhang, Xi Wang et al. · Vision
Continual learning enables vision-language models to accumulate knowledge and adapt to evolving tasks without retraining from scratch. However, in multi-domain task-incremental learning, large domain shifts intensify the stability-plasticity dilemma.
- 2026-05arXiv 2605.06665UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
Minbin Huang, Han Shi, Chuanyang Zheng et al. · Machine learning
Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-to-layer allocation that is then executed privately.
- 2026-04arXiv 2604.18473Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia et al. · Machine learning
Extending a fully post-trained language model with new domain capabilities is fundamentally limited by monolithic training paradigms: retraining from scratch is expensive and scales poorly, while continued training often degrades existing capabilities.
- 2026-04arXiv 2604.06542Does a Global Perspective Help Prune Sparse MoEs Elegantly?
Zeliang Zhang, Nikhil Ghosh, Jiani Liu et al. · Language models & NLP
Empirical scaling laws for language models have encouraged the development of ever-larger LLMs, despite their growing computational and memory costs.
- 2026-03arXiv 2603.19149Optimal Splitting of Language Models from Mixtures to Specialized Domains
Skyler Seto, Pierre Ablin, Anastasiia Filippova et al. · Language models & NLP
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available.
- 2026-03arXiv 2603.06003EvoESAP: Non-Uniform Expert Pruning for Sparse MoE
Zongfang Liu, Shengkun Tang, Boyang Sun et al. · Machine learning
Sparse Mixture-of-Experts (SMoE) language models achieve strong capability at low per-token compute, yet deployment remains constrained by memory footprint and throughput because the full expert pool must still be stored and served.
- 2026-02arXiv 2602.12587Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Anrui Chen, Ruijun Huang, Xin Zhang et al. · Machine learning
Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization.
- 2026-02arXiv 2602.11686LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
Xinyi Liu, Yujie Wang, Fangcheng Fu et al. · Other
Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different input data.
- 2026-02arXiv 2602.09316Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density
Zhendong Mi, Yixiao Chen, Pu Zhao et al. · Machine learning
Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment.
- 2026-02arXiv 2602.03473Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
Meng Lou, Yunxiang Fu, Yizhou Yu · Machine learning
Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years.
- 2026-01arXiv 2601.17616Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
Fatema Siddika, Md Anwar Hossen, Tanwi Mallick et al. · Machine learning
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge.
- 2025-12arXiv 2512.23017Merge before Forget: A Single LoRA Continual Learning via Continual Merging
Fuli Qiao, Mehrdad Mahdavi · Machine learning
Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks.
- 2025-11arXiv 2512.08960Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces
Yueer Zhou, Yichen Wu, Ying Wei · Machine learning
Low-Rank Adaptation (LoRA) enables efficient Continual Learning but often suffers from catastrophic forgetting due to destructive interference between tasks.
- 2025-11arXiv 2511.18987Dynamic Mixture of Experts Against Severe Distribution Shifts
Donghu Kim · Machine learning
The challenge of building neural networks that can continuously learn and adapt to evolving data streams is central to the fields of continual learning (CL) and reinforcement learning (RL).
- 2025-11arXiv 2511.06237Mixtures of SubExperts for Large Language Continual Learning
Haeyong Kang, Hee Suk Yoon, Dahua Feng et al. · Machine learning
, models must incorporate new knowledge without overwriting prior representations) while maintaining scalability under bounded parameter growth.
- 2025-10arXiv 2510.23671Sparsity and Superposition in Mixture of Experts
Marmik Chaudhari, Jeremi Nuer, Rome Thorstenson · Machine learning
Mixture of Experts (MoE) models have become central to scaling large language models, yet their mechanistic differences from dense networks remain poorly understood.
- 2025-10arXiv 2510.13999REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
Mike Lasby, Ivan Lazarevich, Nish Sinnadurai et al. · Machine learning
Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression.
- 2025-10arXiv 2510.08008Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
Ruizhe Wang, Yucheng Ding, Xiao Liu et al. · Machine learning
As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical.
- 2025-08arXiv 2508.07738Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning
Jialu Zhou, Dianxi Shi, Shaowu Yang et al. · Machine learning
Multi-Domain Continual Learning (MDCL) acquires knowledge from sequential tasks with shifting class sets and distribution. Despite the Parameter-Efficient Fine-Tuning (PEFT) methods can adapt for this dual heterogeneity, they still suffer from catastrophic forgetting and forward forgetting.
- 2025-08arXiv 2508.08165Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning
Yan Wang, Da-Wei Zhou, Han-Jia Ye · Vision
Class-Incremental Learning (CIL) requires a learning system to continually learn new classes without forgetting. Existing pre-trained model-based CIL methods often freeze the pre-trained network and adapt to incremental tasks using additional lightweight modules such as adapters.
- 2025-06arXiv 2506.23266Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
Lujun Li, Zhu Qiyuan, Jiacheng Wang et al. · Machine learning
Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges.
- 2025-06arXiv 2506.21035Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts
Haodong Lu, Chongyang Zhao, Minhui Xue et al. · Machine learning
Continual learning (CL) with large pre-trained models aims to incrementally acquire knowledge without catastrophic forgetting.
- 2025-06arXiv 2506.18349SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
Zichong Li, Chen Liang, Zixuan Zhang et al. · Machine learning
The Mixture of Experts (MoE) architecture has emerged as a powerful paradigm for scaling large language models (LLMs) while maintaining inference efficiency. However, their enormous memory requirements make them prohibitively expensive to fine-tune or deploy in resource-constrained environments.
- 2025-06arXiv 2506.01656Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
Ryotaro Kawata, Kohsei Matsutani, Yuri Kinoshita et al. · Machine learning
Mixture of Experts (MoE), an ensemble of specialized models equipped with a router that dynamically distributes each input to appropriate experts, has achieved successful results in the field of machine learning.
- 2025-05arXiv 2505.22389Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Haomiao Qiu, Miao Zhang, Ziyue Qiao et al. · Machine learning
Continual Learning (CL) aims to enable models to continuously acquire new knowledge from a sequence of tasks with avoiding the forgetting of learned information.
- 2025-05arXiv 2506.11038MoTE: Mixture of Task-specific Experts for Pre-Trained ModelBased Class-incremental Learning
Linjie Li, Zhenyu Wu, Yang Ji · Machine learning
Class-incremental learning (CIL) requires deep learning models to continuously acquire new knowledge from streaming data while preserving previously learned information.
- 2025-03arXiv 2503.05029Continual Pre-training of MoEs: How robust is your router?
Benjamin Thérien, Charles-Étienne Joseph, Zain Sarwar et al. · Machine learning
Sparsely-activated Mixture of Experts (MoE) transformers are promising architectures for foundation models.
- 2025-03arXiv 2503.07639Mixture of Experts Made Intrinsically Interpretable
Xingyi Yang, Constantin Venhoff, Ashkan Khakzar et al. · Machine learning
Neurons in large language models often exhibit \emph{polysemanticity}, simultaneously encoding multiple unrelated concepts and obscuring interpretability.
- 2025-02arXiv 2502.19261Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
Taishi Nakamura, Takuya Akiba, Kazuki Fujii et al. · Language models & NLP
The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that initializes and trains an MoE model using a pre-trained dense model.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
