NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2026-11arXiv 2611.01704MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference
Renjie Wei, Haochen Huang, Chuyu Qiu et al. · Machine learning
Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) since it reduces the computational cost through sparse expert activation.
- 2026-08arXiv 2608.08650The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Jiguo Li · Language models & NLP
Mixture-of-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone.
- 2026-05arXiv 2605.31268Mellum2 Technical Report
Marko Kojic, Ivan Bondyrev, Aral de Moor et al. · Language models & NLP
5B active parameters per token.
- 2026-05arXiv 2605.08292Hierarchical Mixture-of-Experts with Two-Stage Optimization
Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov · Machine learning
Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse.
- 2026-04arXiv 2604.23108Mixture of Heterogeneous Grouped Experts for Language Modeling
Zhicheng Ma, Xiang Liu, Zhaoxiang Liu et al. · Language models & NLP
Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently.
- 2026-04arXiv 2604.14419Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
Ivan Ternovtsii, Yurii Bilak · AI & agents
Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating.
- 2026-03arXiv 2603.13364FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
Ning Liao, Xiaoxing Wang, Xiaohan Qin et al. · Vision
As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design.
- 2026-01arXiv 2601.10159What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models
Guimin Hu, Meng Li, Qiwei Peng et al. · Language models & NLP
Most interpretability work focuses on layer- or neuron-level mechanisms in Transformers, leaving expert-level behavior in MoE LLMs underexplored. Motivated by functional specialization in the human brain, we analyze expert activation by distinguishing domain and driver experts.
- 2025-11arXiv 2511.18534HiFi-MambaV2: Hierarchical Shared-Routed MoE for High-Fidelity MRI Reconstruction
Pengcheng Fang, Hongli Chen, Guangzhen Yao et al. · Vision
Reconstructing high-fidelity MR images from undersampled k-space data requires recovering high-frequency details while maintaining anatomical coherence.
- 2025-10arXiv 2510.04136MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Umberto Cappellazzo, Minsu Kim, Pingchuan Ma et al. · Other
Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings.
- 2025-09arXiv 2509.26520Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
Yaoxiang Wang, Qingguo Hu, Yucheng Ding et al. · Language models & NLP
Mixture-of-Experts (MoE) has emerged as a promising paradigm for efficiently scaling large language models without a proportional increase in computational cost.
- 2025-09arXiv 2509.23678Towards a Comprehensive Scaling Law of Mixture-of-Experts
Guoliang Zhao, Yuhan Fu, Shuaipeng Li et al. · Machine learning
Mixture-of-Experts (MoE) models have become the consensus approach for enabling parameter-efficient scaling and cost-effective deployment in large language models.
- 2025-09arXiv 2510.02345Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
Peijun Zhu, Ning Yang, Baoliang Tian et al. · Language models & NLP
Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead. We introduce a unified framework based on dynamic expert clustering and structured compression to address these issues cohesively.
- 2025-07arXiv 2507.17702Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
Changxin Tian, Kunlong Chen, Jia Liu et al. · Language models & NLP
Mixture-of-Experts (MoE) has become a dominant architecture for scaling Large Language Models (LLMs) efficiently by decoupling total parameters from computational cost.
- 2025-07arXiv 2507.11181Mixture of Experts in Large Language Models
Danyang Zhang, Junhao Song, Ziqian Bi et al. · Machine learning
This paper presents a comprehensive review of the Mixture-of-Experts (MoE) architecture in large language models, highlighting its ability to significantly enhance model performance while maintaining minimal computational overhead.
- 2025-07arXiv 2507.10524Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
Sangmin Bae, Yujin Kim, Reza Bayat et al. · Language models & NLP
Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive.
- 2025-06arXiv 2506.12119Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
Houyi Li, Ka Man Lo, Shijie Xuyang et al. · Language models & NLP
Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute.
- 2025-05arXiv 2505.14173THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
Yunlong Liang, Fandong Meng, Jie Zhou · Language models & NLP
The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT).
- 2025-05arXiv 2505.09388Qwen3 Technical Report
An Yang, Anfeng Li, Baosong Yang et al. · Language models & NLP
In this work, we present Qwen3, the latest version of the Qwen model family. Qwen3 comprises a series of large language models (LLMs) designed to advance performance, efficiency, and multilingual capabilities.
- 2025-04arXiv 2504.06426S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning
Hanqing Zeng, Yinglong Xia, Zhuokai Zhao et al. · Language models & NLP
Fine-tuning pre-trained large language models (LLMs) presents a dual challenge of balancing parameter efficiency and model capacity.
- 2025-03arXiv 2503.07137A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications
Siyuan Mu, Sen Lin · Machine learning
Artificial intelligence (AI) has achieved astonishing successes in many domains, especially with the recent breakthroughs in the development of foundational large models.
- 2025-02arXiv 2502.10928Probing Semantic Routing in Large Mixture-of-Expert Models
Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck et al. · Machine learning
In the past year, large (>100B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of efficiency, prior work has also explored functional differentiation through routing behavior.
- 2025-02arXiv 2502.05172Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski et al. · Machine learning
Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models.
- 2024-12arXiv 2412.19437DeepSeek-V3 Technical Report
DeepSeek-AI, Aixin Liu, Bei Feng et al. · Language models & NLP
We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.
- 2024-09arXiv 2409.19878HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models
Bingshen Mu, Kun Wei, Qijie Shao et al. · Other
Recent advancements in integrating Large Language Models (LLM) with automatic speech recognition (ASR) have performed remarkably in general domains.
- 2024-09arXiv 2409.02060OLMoE: Open Mixture-of-Experts Language Models
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld et al. · Language models & NLP
We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token.
- 2024-07arXiv 2407.21770MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
Xi Victoria Lin, Akshat Shrivastava, Liang Luo et al. · AI & agents
We introduce MoMa, a novel modality-aware mixture-of-experts (MoE) architecture designed for pre-training mixed-modal, early-fusion language models. MoMa processes images and text in arbitrary sequences by dividing expert modules into modality-specific groups.
- 2024-07arXiv 2407.19985Mixture of Nested Experts: Adaptive Processing of Visual Tokens
Gagan Jain, Nidhi Hegde, Aditya Kusupati et al. · Vision
The visual medium (images and videos) naturally contains a large amount of information redundancy, thereby providing a great opportunity for leveraging efficiency in processing.
- 2024-07arXiv 2407.18581Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing
Hukai Huang, Shenghui Lu, Yahui Shan et al. · Language models & NLP
The Mixture of Experts (MoE) model is a promising approach for handling code-switching speech recognition (CS-ASR) tasks. However, the existing CS-ASR work on MoE has yet to leverage the advantages of MoE's parameter scaling ability fully.
- 2024-07arXiv 2407.10671Qwen2 Technical Report
An Yang, Baosong Yang, Binyuan Hui et al. · Language models & NLP
This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models.5 to 72 billion, featuring dense models and a Mixture-of-Experts model.
- 2024-07arXiv 2407.09816MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts
Zhenpeng Su, Zijia Lin, Xue Bai et al. · Language models & NLP
Scaling the size of a model enhances its capabilities but significantly increases computation complexity. Mixture-of-Experts models (MoE) address the issue by allowing model size to scale up without substantially increasing training or inference costs.
- 2024-07arXiv 2407.04153Mixture of A Million Experts
Xu Owen He · Machine learning
The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows.
- 2024-06arXiv 2406.18665RouteLLM: Learning to Route LLMs with Preference Data
Isaac Ong, Amjad Almahairi, Vincent Wu et al. · Machine learning
Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost.
Cited inRoute on shared statistics, never expert confidence: the router architecture for frozen expertsStudy: Hierarchical and T shaped MoEInternal notesStudy: Model self knowledge and confident errorsStudy: Optimal router architectureStudy: Orchestrated specialists and bridge synthesisStudy: Stopping confident wrong expert votes - 2024-06arXiv 2406.04692Mixture-of-Agents Enhances Large Language Model Capabilities
Junlin Wang, Jue Wang, Ben Athiwaratkun et al. · Language models & NLP
Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction.
- 2024-05arXiv 2405.16039MoEUT: Mixture-of-Experts Universal Transformers
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber et al. · Machine learning
Previous work on Universal Transformers (UTs) has demonstrated the importance of parameter sharing across layers.
- 2024-05arXiv 2405.04434DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Aixin Liu, Bei Feng et al. · Language models & NLP
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens.
- 2024-05arXiv 2405.03133Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
Zexuan Zhong, Mengzhou Xia, Danqi Chen et al. · Language models & NLP
Mixture-of-experts (MoE) models facilitate efficient scaling; however, training the router network introduces the challenge of optimizing a non-differentiable, discrete objective.
- 2024-04arXiv 2404.15045Multi-Head Mixture-of-Experts
Xun Wu, Shaohan Huang, Wenhui Wang et al. · Language models & NLP
Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for optimization.
- 2024-04arXiv 2404.13628Mixture of LoRA Experts
Xun Wu, Shaohan Huang, Furu Wei · Language models & NLP
LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques.
- 2024-04arXiv 2404.02258Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
David Raposo, Sam Ritter, Blake Richards et al. · Machine learning
Transformer-based language models spread FLOPs uniformly across input sequences.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
