NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2021-10arXiv 2110.06296The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
Rahim Entezari, Hanie Sedghi, Olga Saukh et al. · Machine learning
In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them.
- 2021-10arXiv 2110.01786MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu et al. · Language models & NLP
Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear.
- 2021-02arXiv 2102.08574Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks
Lemeng Wu, Bo Liu, Peter Stone et al. · Machine learning
We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures.
- 2021-01arXiv 2101.03961Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
William Fedus, Barret Zoph, Noam Shazeer · Machine learning
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example.
- 2020-03arXiv 2003.10392Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting
Lemeng Wu, Mao Ye, Qi Lei et al. · Machine learning
Developing efficient and principled neural architecture optimization methods is a critical challenge of modern deep learning.
- 2020-01arXiv 2001.00689A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
Soochan Lee, Junsoo Ha, Dongsu Zhang et al. · Machine learning
Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable, and task boundaries are known during training.
- 2019-12arXiv 1912.05671Linear Mode Connectivity and the Lottery Ticket Hypothesis
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy et al. · Machine learning
, random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region.
- 2019-10arXiv 1910.03103Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent
Dilin Wang, Meng Li, Lemeng Wu et al. · Machine learning
Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited.
- 2019-10arXiv 1910.02366Splitting Steepest Descent for Growing Neural Architectures
Qiang Liu, Lemeng Wu, Dilin Wang · Machine learning
We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs.
- 2017-11arXiv 1711.05769PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
Arun Mallya, Svetlana Lazebnik · Vision
This paper presents a method for adding multiple tasks to a single deep neural network while avoiding catastrophic forgetting. Inspired by network pruning techniques, we exploit redundancies in large deep networks to free up parameters that can then be employed to learn new tasks.
- 2017-08arXiv 1708.01547Lifelong Learning with Dynamically Expandable Networks
Jaehong Yoon, Eunho Yang, Jeongtae Lee et al. · Machine learning
We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks.
- 2016-11arXiv 1611.06194Expert Gate: Lifelong Learning with a Network of Experts
Rahaf Aljundi, Punarjay Chakravarty, Tinne Tuytelaars · Vision
In this paper we introduce a model of lifelong learning, based on a Network of Experts. New tasks / experts are learned and added to the model sequentially, building on what was learned before.
- 2016-06arXiv 1606.04671Progressive Neural Networks
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins et al. · Machine learning
Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence.
- 2015-11arXiv 1511.05641Net2Net: Accelerating Learning via Knowledge Transfer
Tianqi Chen, Ian Goodfellow, Jonathon Shlens · Machine learning
We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net.
- 2011-11arXiv 1111.0352Revisiting k-means: New Algorithms via Bayesian Nonparametrics
Brian Kulis, Michael I. Jordan · Machine learning
Bayesian models offer great flexibility for clustering applications---Bayesian nonparametrics can be used for modeling infinite mixtures, and hierarchical Bayesian models can be utilized for sharing clusters across multiple data sets.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
