NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2024-03arXiv 2403.07816Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma et al. · Language models & NLP
We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge.
Cited inRoute on shared statistics, never expert confidence: the router architecture for frozen expertsStudy: Fortifying knowledge in continual learningStudy: Hierarchical and T shaped MoEStudy: Merging and splitting MoE expertsStudy: Optimal router architectureStudy: Orchestrated specialists and bridge synthesis - 2024-02arXiv 2402.07871Scaling Laws for Fine-Grained Mixture of Experts
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski et al. · Machine learning
Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables.
- 2024-01arXiv 2401.06066DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Damai Dai, Chengqi Deng, Chenggang Zhao et al. · Language models & NLP
In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. each expert acquires non-overlapping and focused knowledge.
- 2024-01arXiv 2401.04088Mixtral of Experts
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux et al. · Machine learning
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs.
- 2023-10arXiv 2310.12963AutoMix: Automatically Mixing Language Models
Pranjal Aggarwal, Aman Madaan, Ankit Anand et al. · Language models & NLP
Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging.
- 2023-08arXiv 2308.00951From Sparse to Soft Mixtures of Experts
Joan Puigcerver, Carlos Riquelme, Basil Mustafa et al. · Machine learning
Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning.
- 2023-07arXiv 2307.02764When Does Confidence-Based Cascade Deferral Suffice?
Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon et al. · Machine learning
Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction.
- 2023-06arXiv 2306.17165An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training
Zitian Chen, Mingyu Ding, Yikang Shen et al. · Vision
We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Despite considerable progress in multi-task learning, most efforts focus on learning from multi-label data: a single image set with multiple task labels.
- 2022-12arXiv 2212.05055Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp et al. · Machine learning
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks.
- 2022-11arXiv 2211.15259A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification
Paul F. Jaeger, Carsten T. Lüth, Lukas Klein et al. · ICLR 2023 (oral) · Vision
Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores.
- 2022-11arXiv 2211.07725Hierarchically Structured Task-Agnostic Continual Learning
Heinke Hihn, Daniel A. Braun · Machine learning
One notable weakness of current machine learning algorithms is the poor ability of models to solve new problems without forgetting previously acquired knowledge.
- 2022-04arXiv 2204.08396StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma et al. · Machine learning
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference.
- 2022-02arXiv 2202.09368Mixture-of-Experts with Expert Choice Routing
Yanqi Zhou, Tao Lei, Hanxiao Liu et al. · Machine learning
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged.
- 2022-02arXiv 2202.08906ST-MoE: Designing Stable and Transferable Sparse Expert Models
Barret Zoph, Irwan Bello, Sameer Kumar et al. · Language models & NLP
Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models.
- 2021-06arXiv 2106.04426Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam et al. · Machine learning
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence.
- 2021-03arXiv 2103.16716BASE Layers: Simplifying Training of Large, Sparse Models
Mike Lewis, Shruti Bhosale, Tim Dettmers et al. · Language models & NLP
We introduce a new balanced assignment of experts (BASE) layer for large language models that greatly simplifies existing high capacity sparse layers.
- 2021-01arXiv 2101.03961Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
William Fedus, Barret Zoph, Noam Shazeer · Machine learning
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example.
- 2020-06arXiv 2006.16668GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu et al. · Language models & NLP
Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data and compute.
- 2020-06arXiv 2006.01862Consistent Estimators for Learning to Defer to an Expert
Hussein Mozannar, David Sontag · Machine learning
Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms.
- 2017-01arXiv 1701.06538Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz et al. · Machine learning
The capacity of a neural network to absorb information is limited by its number of parameters.
- 2015-06arXiv 1506.01567Classification with many classes: challenges and pluses
Felix Abramovich, Marianna Pensky · Statistics & theory
The objective of the paper is to study accuracy of multi-class classification in high-dimensional setting, where the number of classes is also large ("large , large , small " model).
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
