NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2024-03arXiv 2403.08763Simple and Scalable Strategies to Continually Pre-train Large Language Models
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta et al. · Machine learning
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training.
Cited inTo keep what a model knows, control what it is allowed to moveStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Hardware guardian core for continual learningStudy: Hierarchical and T shaped MoEStudy: Orchestrated specialists and bridge synthesis - 2024-03arXiv 2403.07816Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma et al. · Language models & NLP
We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge.
Cited inRoute on shared statistics, never expert confidence: the router architecture for frozen expertsStudy: Fortifying knowledge in continual learningStudy: Hierarchical and T shaped MoEStudy: Merging and splitting MoE expertsStudy: Optimal router architectureStudy: Orchestrated specialists and bridge synthesis - 2024-02arXiv 2402.07871Scaling Laws for Fine-Grained Mixture of Experts
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski et al. · Machine learning
Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables.
- 2024-01arXiv 2402.01739OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Fuzhao Xue, Zian Zheng, Yao Fu et al. · Language models & NLP
To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens.
- 2024-01arXiv 2401.06066DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Damai Dai, Chengqi Deng, Chenggang Zhao et al. · Language models & NLP
In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. each expert acquires non-overlapping and focused knowledge.
- 2024-01arXiv 2401.04088Mixtral of Experts
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux et al. · Machine learning
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs.
- 2023-12arXiv 2312.09979LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
Shihan Dou, Enyu Zhou, Yan Liu et al. · Language models & NLP
Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks.
- 2023-11arXiv 2311.16452Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Harsha Nori, Yin Tat Lee, Sheng Zhang et al. · Language models & NLP
Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models.
- 2023-11arXiv 2311.10770Exponentially Faster Language Modelling
Peter Belcak, Roger Wattenhofer · Language models & NLP
Language models only really need to use an exponential fraction of their neurons for individual inferences.3% of its neurons during inference while performing on par with similar BERT models.
- 2023-11arXiv 2311.03099Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
Le Yu, Bowen Yu, Haiyang Yu et al. · Language models & NLP
In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs.
- 2023-08arXiv 2308.14711Fast Feedforward Networks
Peter Belcak, Roger Wattenhofer · Machine learning
We break the linear link between the layer size and its inference cost by introducing the fast feedforward (FFF) architecture, a log-time alternative to feedforward networks.
- 2023-06arXiv 2306.01708TIES-Merging: Resolving Interference When Merging Models
Prateek Yadav, Derek Tam, Leshem Choshen et al. · Machine learning
, further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency.
Cited inStudy: Decision signatures for conflict detectionStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learningStudy: Hierarchical and T shaped MoEStudy: LLMs modifying open model weightsStudy: Merging and splitting MoE expertsStudy: Weight editing to prevent forgetting - 2023-05arXiv 2305.05862Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks
Xianzhi Li, Samuel Chan, Xiaodan Zhu et al. · Language models & NLP
The most recent large language models(LLMs) such as ChatGPT and GPT-4 have shown exceptional capabilities of generalist models, achieving state-of-the-art performance on a wide range of NLP tasks with little or no adaptation.
- 2023-05arXiv 2305.05176FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Lingjiao Chen, Matei Zaharia, James Zou · Machine learning
There is a rapidly growing number of large language models (LLMs) that users can query for a fee.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude.
- 2023-03arXiv 2303.17564BloombergGPT: A Large Language Model for Finance
Shijie Wu, Ozan Irsoy, Steven Lu et al. · Machine learning
The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering.
- 2023-03arXiv 2303.17580HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Yongliang Shen, Kaitao Song, Xu Tan et al. · Language models & NLP
Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence. While there are numerous AI models available for various domains and modalities, they cannot handle complicated AI tasks autonomously.
- 2023-03arXiv 2303.10845PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng et al. · Language models & NLP
The scaling of large language models has greatly improved natural language understanding, generation, and reasoning.085T parameters named PanGu-Σ.
- 2022-12arXiv 2212.05055Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp et al. · Machine learning
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks.
- 2022-12arXiv 2212.04089Editing Models with Task Arithmetic
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman et al. · Machine learning
, improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems.
- 2022-08arXiv 2208.03306Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
Margaret Li, Suchin Gururangan, Tim Dettmers et al. · Language models & NLP
We present Branch-Train-Merge (BTM), a communication-efficient algorithm for embarrassingly parallel training of large language models (LLMs).
- 2022-02arXiv 2202.09368Mixture-of-Experts with Expert Choice Routing
Yanqi Zhou, Tao Lei, Hanxiao Liu et al. · Machine learning
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged.
- 2022-02arXiv 2202.08906ST-MoE: Designing Stable and Transferable Sparse Expert Models
Barret Zoph, Irwan Bello, Sameer Kumar et al. · Language models & NLP
Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models.
- 2022-02arXiv 2202.01169Unified Scaling Laws for Routed Language Models
Aidan Clark, Diego de las Casas, Aurelia Guy et al. · Language models & NLP
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input.
- 2022-01arXiv 2201.05596DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Samyam Rajbhandari, Conglong Li, Zhewei Yao et al. · Machine learning
As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model.
- 2021-06arXiv 2106.04426Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam et al. · Machine learning
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence.
- 2020-11arXiv 2011.10605Nested Mixture of Experts: Cooperative and Competitive Learning of Hybrid Dynamical System
Junhyeok Ahn, Luis Sentis · Machine learning
Model-based reinforcement learning (MBRL) algorithms can attain significant sample efficiency but require an appropriate network structure to represent system dynamics.
- 2019-07arXiv 1907.05242Large Memory Layers with Product Keys
Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato et al. · Language models & NLP
This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead.
- 2019-03arXiv 1903.07756Hierarchical Routing Mixture of Experts
Wenbo Zhao, Yang Gao, Shahan Ali Memon et al. · Machine learning
In regression tasks the distribution of the data is often too complex to be fitted by a single model. In contrast, partition-based models are developed where data is divided and fitted by local models.
- 2019-02arXiv 1902.00751Parameter-Efficient Transfer Learning for NLP
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski et al. · Machine learning
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task.
- 2017-07arXiv 1707.04227Automatic Speech Recognition with Very Large Conversational Finnish and Estonian Vocabularies
Seppo Enarvi, Peter Smit, Sami Virpioja et al. · IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 11, pp. 2085-2097, November 2017 · Language models & NLP
Today, the vocabulary size for language models in large vocabulary speech recognition is typically several hundreds of thousands of words. While this is already sufficient in some applications, the out-of-vocabulary words are still limiting the usability in others.
- 2017-01arXiv 1701.06538Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz et al. · Machine learning
The capacity of a neural network to absorb information is limited by its number of parameters.
- 2015-12arXiv 1512.04906Strategies for Training Large Vocabulary Neural Language Models
Welin Chen, David Grangier, Michael Auli · Language models & NLP
Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney.
- 2015-05arXiv 1505.06353The evolutionary origins of hierarchy
Henok Mengistu, Joost Huizinga, Jean-Baptiste Mouret et al. · Neuroscience & neural computation
Hierarchical organization -- the recursive composition of sub-modules -- is ubiquitous in biological networks, including neural, metabolic, ecological, and genetic regulatory networks, and in human-made systems, such as large organizations and the Internet.
- 2013-12arXiv 1312.4314Learning Factored Representations in a Deep Mixture of Experts
David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever · Machine learning
Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space. This is achieved by training a "gating" network that maps each input to a distribution over the experts.
- 2013-02arXiv 1302.4267Mutation Rules and the Evolution of Sparseness and Modularity in Biological Systems
Tamar Friedlander, Avraham E. Mayo, Tsvi Tlusty et al. · PLoS ONE 8(8): e70444 · Neuroscience & neural computation
Biological systems exhibit two structural features on many levels of organization: sparseness, in which only a small fraction of possible interactions between components actually occur; and modularity - the near decomposability of the system into modules with distinct functionality.
- 2012-10arXiv 1212.2447Bayesian Hierarchical Mixtures of Experts
Christopher M. Bishop, Markus Svensen · Machine learning
The Hierarchical Mixture of Experts (HME) is a well-known tree-based model for regression and classification, based on soft probabilistic splits. In its original formulation it was trained by maximum likelihood, and is therefore prone to over-fitting.
- 2012-07arXiv 1207.2743The evolutionary origins of modularity
Jeff Clune, Jean-Baptiste Mouret, Hod Lipson · Clune J, Mouret J-B, Lipson H. 2013 The evolutionary origins of modularity. Proceedings of the Royal Society B. 280: 20122863 · Neuroscience & neural computation
A central biological question is how natural organisms are so evolvable (capable of quickly adapting to new environments).
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
