NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2020-04arXiv 2004.09124Compositionality and Generalization in Emergent Languages
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt et al. · Language models & NLP
Natural language allows us to refer to novel composite concepts by combining expressions denoting their parts according to systematic rules, a property known as \emph{compositionality}.
Cited inWhat Language Buys - 2020-04arXiv 2004.07211Dark Experience for General Continual Learning: a Strong, Simple Baseline
Pietro Buzzega, Matteo Boschini, Angelo Porrello et al. · Machine learning
Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable.
- 2020-04arXiv 2004.03473Learning from Imperfect Annotations
Emmanouil Antonios Platanios, Maruan Al-Shedivat, Eric Xing et al. · Machine learning
Many machine learning systems today are trained on large amounts of human-annotated data. Data annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases.
- 2020-03arXiv 2003.11942Towards Backward-Compatible Representation Learning
Yantao Shen, Yuanjun Xiong, Wei Xia et al. · Vision
We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions and are learned via different neural network architectures and loss functions.
- 2020-03arXiv 2003.10392Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting
Lemeng Wu, Mao Ye, Qi Lei et al. · Machine learning
Developing efficient and principled neural architecture optimization methods is a critical challenge of modern deep learning.
- 2020-02arXiv 2002.10689A Theory of Usable Information Under Computational Constraints
Yilun Xu, Shengjia Zhao, Jiaming Song et al. · Machine learning
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer.
Cited inWhat Language Buys - 2020-02arXiv 2002.08484Estimating Training Data Influence by Tracing Gradient Descent
Garima Pruthi, Frederick Liu, Mukund Sundararajan et al. · Machine learning
We introduce a method called TracIn that computes the influence of a training example on a prediction made by the model. The idea is to trace how the loss on the test point changes during the training process whenever the training example of interest was utilized.
- 2020-02arXiv 2002.06673Performative Prediction
Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner et al. · Machine learning
When predictions support decisions they may influence the outcome they aim to predict. We call such predictions performative; the prediction influences the target.
- 2020-02arXiv 2002.05969Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings
Hongyu Ren, Weihua Hu, Jure Leskovec · Machine learning
Answering complex logical queries on large-scale incomplete knowledge graphs (KGs) is a fundamental yet challenging task.
- 2020-02arXiv 2002.01365Compositional Languages Emerge in a Neural Iterated Learning Model
Yi Ren, Shangmin Guo, Matthieu Labeau et al. · ICLR-2020 · Language models & NLP
The principle of compositionality, which enables natural language to represent complex concepts via a structured combination of simpler ones, allows us to convey an open-ended set of messages using a limited vocabulary.
- 2020-01arXiv 2001.08361Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan et al. · Machine learning
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.
- 2020-01arXiv 2001.06782Gradient Surgery for Multi-Task Learning
Tianhe Yu, Saurabh Kumar, Abhishek Gupta et al. · Machine learning
While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge.
- 2020-01arXiv 2001.00689A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
Soochan Lee, Junsoo Ha, Dongsu Zhang et al. · Machine learning
Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable, and task boundaries are known during training.
- 2019-12arXiv 1912.05671Linear Mode Connectivity and the Lottery Ticket Hypothesis
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy et al. · Machine learning
, random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region.
- 2019-11arXiv 1911.04933Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Aditya Golatkar, Alessandro Achille, Stefano Soatto · Machine learning
We explore the problem of selectively forgetting a particular subset of the data used for training a deep neural network. While the effects of the data to be forgotten can be hidden from the output of the network, insights may still be gleaned by probing deep into its weights.
- 2019-11arXiv 1911.01547On the Measure of Intelligence
François Chollet · AI & agents
To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans.
Cited inStudy: Superintelligence - 2019-10arXiv 1910.08348VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl et al. · Machine learning
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment.
Cited inStudy: Genesys program prior art - 2019-10arXiv 1910.08475On Warm-Starting Neural Network Training
Jordan T. Ash, Ryan P. Adams · 2020 Advances in Neural Information Processing Systems · Machine learning
In many real-world deployments of machine learning systems, data arrive piecemeal., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data.
- 2019-10arXiv 1910.07224Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
Rémy Portelas, Cédric Colas, Katja Hofmann et al. · Machine learning
We consider the problem of how a teacher algorithm can enable an unknown Deep Reinforcement Learning (DRL) student to become good at a skill over a wide range of diverse environments.
- 2019-10arXiv 1910.07104Orthogonal Gradient Descent for Continual Learning
Mehrdad Farajtabar, Navid Azizan, Alex Mott et al. · Machine learning
Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains.
- 2019-10arXiv 1910.03103Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent
Dilin Wang, Meng Li, Lemeng Wu et al. · Machine learning
Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited.
- 2019-10arXiv 1910.02366Splitting Steepest Descent for Growing Neural Architectures
Qiang Liu, Lemeng Wu, Dilin Wang · Machine learning
We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs.
- 2019-10arXiv 1910.01348On the Efficacy of Knowledge Distillation
Jang Hyun Cho, Bharath Hariharan · ICCV 2019 · Machine learning
In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures.
- 2019-09arXiv 1909.03329LAMOL: LAnguage MOdeling for Lifelong Language Learning
Fan-Keng Sun, Cheng-Hao Ho, Hung-Yi Lee · Language models & NLP
Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling.
- 2019-09arXiv 1909.02187Adaptive and Efficient Algorithms for Tracking the Best Expert
Shiyin Lu, Lijun Zhang · Machine learning
In this paper, we consider the problem of prediction with expert advice in dynamic environments. We choose tracking regret as the performance metric and develop two adaptive and efficient algorithms with data-dependent tracking regret bounds.
- 2019-09arXiv 1909.01377Deep Equilibrium Models
Shaojie Bai, J. Zico Kolter, Vladlen Koltun · Machine learning
We present a new approach to modeling sequential data: the deep equilibrium model (DEQ).
- 2019-08arXiv 1908.11775Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada et al. · Machine learning
Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction.
- 2019-08arXiv 1908.04734Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective
Tom Everitt, Marcus Hutter, Ramana Kumar et al. · AI & agents
Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal?
- 2019-08arXiv 1908.02240Biologically inspired sleep algorithm for artificial neural networks
Giri P Krishnan, Timothy Tadros, Ramyaa Ramyaa et al. · Neuroscience & neural computation
Sleep plays an important role in incremental learning and consolidation of memories in biological systems.
- 2019-07arXiv 1907.05242Large Memory Layers with Product Keys
Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato et al. · Language models & NLP
This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead.
- 2019-07arXiv 1907.03822Graph Policy Gradients for Large Scale Robot Control
Arbaaz Khan, Ekaterina Tolstaya, Alejandro Ribeiro et al. · Robotics & control
In this paper, we consider the problem of learning policies to control a large number of homogeneous robots. To this end, we propose a new algorithm we call Graph Policy Gradients (GPG) that exploits the underlying graph symmetry among the robots.
Cited inStudy: Arvind Gupta meeting prep - 2019-06arXiv 1907.00208Deep Gamblers: Learning to Abstain with Portfolio Theory
Liu Ziyin, Zhikang Wang, Paul Pu Liang et al. · Machine learning
We deal with the \textit{selective classification} problem (supervised-learning problem with a rejection option), where we want to achieve the best performance at a certain level of coverage of the data.
- 2019-06arXiv 1906.04358Weight Agnostic Neural Networks
Adam Gaier, David Ha · Machine learning
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture?
- 2019-06arXiv 1906.02530Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Yaniv Ovadia, Emily Fertig, Jie Ren et al. · Machine learning
Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive {\em uncertainty}.
- 2019-05arXiv 1905.13260Large Scale Incremental Learning
Yue Wu, Yinpeng Chen, Lijuan Wang et al. · Vision
Modern machine learning suffers from catastrophic forgetting when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes.
- 2019-05arXiv 1905.12558Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Frederik Kunstner, Lukas Balles, Philipp Hennig · Machine learning
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
- 2019-05arXiv 1905.11614Uncertainty-based Continual Learning with Adaptive Regularization
Hongjoon Ahn, Sungmin Cha, Donggyu Lee et al. · Machine learning
We introduce a new neural network-based continual learning algorithm, dubbed as Uncertainty-regularized Continual Learning (UCL), which builds on traditional Bayesian online learning framework with variational inference.
- 2019-05arXiv 1905.10985AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
Jeff Clune · AI & agents
Perhaps the most ambitious scientific quest in human history is the creation of general artificial intelligence, which roughly means AI that is as smart or smarter than humans.
- 2019-04arXiv 1904.07734Three scenarios for continual learning
Gido M. van de Ven, Andreas S. Tolias · Machine learning
Standard artificial neural networks suffer from the well-known issue of catastrophic forgetting, making continual or lifelong learning difficult for machine learning.
- 2019-04arXiv 1904.06019Conformal Prediction Under Covariate Shift
Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes et al. · Statistics & theory
We extend conformal prediction methodology beyond the case of exchangeable data.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
