NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2021-02arXiv 2102.08574Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks
Lemeng Wu, Bo Liu, Peter Stone et al. · Machine learning
We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures.
- 2021-02arXiv 2102.06698A dynamic network model of societal complexity and resilience inspired by Tainter's theory of collapse
Florian Schunck, Marc Wiedermann, Jobst Heitzig et al. · Hardware, photonics & physics
In recent years, several global events have severely disrupted economies and social structures, undermining confidence in the resilience of modern societies.
- 2021-02arXiv 2102.06284Large Scale Distributed Collaborative Unlabeled Motion Planning with Graph Policy Gradients
Arbaaz Khan, Vijay Kumar, Alejandro Ribeiro · Robotics & control
In this paper, we present a learning method to solve the unlabelled motion problem with motion constraints and space constraints in 2D space for a large number of robots.
Cited inStudy: Arvind Gupta meeting prep - 2021-02arXiv 2102.03896Consequences of Misaligned AI
Simon Zhuang, Dylan Hadfield-Menell · NeurIPS 2020 · AI & agents
AI systems often rely on two key components: a specified goal or reward function and an optimization algorithm to compute the optimal behavior for that goal. This approach is intended to provide value for a principal: the user on whose behalf the agent acts.
Cited inWhen the Exploit Is Constructed - 2021-02arXiv 2102.01017Measuring and Improving Consistency in Pretrained Language Models
Yanai Elazar, Nora Kassner, Shauli Ravfogel et al. · Language models & NLP
Consistency of a model -- that is, the invariance of its behavior under meaning-preserving alternations in its input -- is a highly desirable property in natural language processing.
- 2021-01arXiv 2101.07592Synaptic metaplasticity in binarized neural networks
Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin et al. · Computational and Systems Neuroscience (Cosyne) 2021 · Neuroscience & neural computation
Unlike the brain, artificial neural networks, including state-of-the-art deep neural networks for computer vision, are subject to "catastrophic forgetting": they rapidly forget the previous task when trained on a new one.
- 2021-01arXiv 2101.03961Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
William Fedus, Barret Zoph, Noam Shazeer · Machine learning
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example.
- 2020-12arXiv 2012.14210The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes
Nils Reimers, Iryna Gurevych · Retrieval & data
Information Retrieval using dense low-dimensional representations recently became popular and showed out-performance to traditional sparse-representations like BM25.
- 2020-12arXiv 2012.02096Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
Michael Dennis, Natasha Jaques, Eugene Vinitsky et al. · Machine learning
A wide range of reinforcement learning (RL) problems - including robustness, transfer learning, unsupervised RL, and emergent complexity - require specifying a distribution of tasks or environments in which a policy will be trained.
- 2020-11arXiv 2011.10605Nested Mixture of Experts: Cooperative and Competitive Learning of Hybrid Dynamical System
Junhyeok Ahn, Luis Sentis · Machine learning
Model-based reinforcement learning (MBRL) algorithms can attain significant sample efficiency but require an appropriate network structure to represent system dynamics.
- 2020-11arXiv 2011.09161Positive-Congruent Training: Towards Regression-Free Model Updates
Sijie Yan, Yuanjun Xiong, Kaustav Kundu et al. · Vision
Reducing inconsistencies in the behavior of different versions of an AI system can be as important in practice as reducing its overall error.
- 2020-10arXiv 2010.15277Class-incremental learning: survey and performance evaluation on image classification
Marc Masana, Xialei Liu, Bartlomiej Twardowski et al. · Machine learning
For future learning systems, incremental learning is desirable because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored -- also important when privacy…
- 2020-10arXiv 2010.11465Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs
Hongyu Ren, Jure Leskovec · AI & agents
One of the fundamental problems in Artificial Intelligence is to perform complex multi-hop logical reasoning over the facts captured by a knowledge graph (KG). This problem is challenging, because KGs can be massive and incomplete.
- 2020-10arXiv 2010.11929An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · Vision
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited.
- 2020-10arXiv 2010.09686Estimating means of bounded random variables by betting
Ian Waudby-Smith, Aaditya Ramdas · Statistics & theory
This paper derives confidence intervals (CI) and time-uniform confidence sequences (CS) for the classical problem of estimating an unknown mean from bounded observations.
Cited inInternal notes - 2020-10arXiv 2010.06808Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
Zhao Chen, Jiquan Ngiam, Yanping Huang et al. · Machine learning
The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights.
- 2020-10arXiv 2010.05874Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Zirui Wang, Yulia Tsvetkov, Orhan Firat et al. · Language models & NLP
Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization.
- 2020-10arXiv 2010.03759Energy-based Out-of-distribution Detection
Weitang Liu, Xiaoyun Wang, John D. Owens et al. · Machine learning
Determining whether inputs are out-of-distribution (OOD) is an essential building block for safely deploying machine learning models in the open world. However, previous methods relying on the softmax confidence score suffer from overconfident posterior distributions for OOD data.
- 2020-10arXiv 2010.04003A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
Thang Doan, Mehdi Bennani, Bogdan Mazoure et al. · Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS 2021) · Machine learning
Continual learning (CL) is a setting in which an agent has to learn from an incoming stream of data during its entire lifetime. Although major advances have been made in the field, one recurring problem which remains unsolved is that of Catastrophic Forgetting (CF).
- 2020-09arXiv 2009.14794Rethinking Attention with Performers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan et al. · Machine learning
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness.
- 2020-07arXiv 2007.08176CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
Jihoon Tack, Sangwoo Mo, Jongheon Jeong et al. · Machine learning
, identifying whether a given sample is drawn from outside the training distribution, is essential for reliable machine learning. To this end, there have been many attempts at learning a representation well-suited for novelty detection and designing a score based on such representation.
- 2020-07arXiv 2008.02217Hopfield Networks is All You Need
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner et al. · Neuroscience & neural computation
We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors.
- 2020-07arXiv 2007.07400Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Vinay V. Ramasesh, Ethan Dyer, Maithra Raghu · Machine learning
A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks.
- 2020-07arXiv 2007.05134Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
Shreyas Padhy, Zachary Nado, Jie Ren et al. · Machine learning
Accurate estimation of predictive uncertainty in modern neural networks is critical to achieve well calibrated predictions and detect out-of-distribution (OOD) inputs.
- 2020-07arXiv 2007.04965A Study on Encodings for Neural Architecture Search
Colin White, Willie Neiswanger, Sam Nolen et al. · Advances in Neural Information Processing Systems 2020 · Machine learning
Neural architecture search (NAS) has been extensively studied in the past few years.
- 2020-07arXiv 2007.02686Meta-Learning through Hebbian Plasticity in Random Networks
Elias Najarro, Sebastian Risi · Advances in Neural Information Processing Systems (2020) · Neuroscience & neural computation
Lifelong learning and adaptability are two defining aspects of biological agents.
- 2020-06arXiv 2006.16668GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu et al. · Language models & NLP
Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data and compute.
- 2020-06arXiv 2006.16236Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas et al. · Machine learning
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences.
- 2020-06arXiv 2006.14769Supermasks in Superposition
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu et al. · Machine learning
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance.
- 2020-06arXiv 2006.14651Influence Functions in Deep Learning Are Fragile
Samyadeep Basu, Philip Pope, Soheil Feizi · Machine learning
Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation.
- 2020-06arXiv 2006.11722Type-II Superconductivity below 4K in Sn0.4Sb0.6
M. M. Sharma, Kapil Kumar, Lina Sang et al. · J. Alloys and Compounds (2020) · Hardware, photonics & physics
6 single crystal at below 4K. Rietveld refined Powder XRD data confirms the phase purity of as grown crystal, crystallizing in rhombohedral R-3m space group with an elongated (2xc) unit cell in c-direction.
- 2020-06arXiv 2006.10701Deep Reinforcement Learning amidst Lifelong Non-Stationarity
Annie Xie, James Harrison, Chelsea Finn · Machine learning
As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives. In contrast, typical reinforcement learning problem set-ups consider decision processes that are stationary across episodes.
Cited inStudy: Genesys program prior art - 2020-06arXiv 2006.10108Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy et al. · Machine learning
Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost.
- 2020-06arXiv 2006.09462Selective Question Answering under Domain Shift
Amita Kamath, Robin Jia, Percy Liang · Language models & NLP
To avoid giving wrong answers, question answering (QA) models need to know when to abstain from answering. Moreover, users often ask questions that diverge from the model's training data, making errors more likely and thus abstention more critical.
- 2020-06arXiv 2006.08591Monotone operator equilibrium networks
Ezra Winston, J. Zico Kolter · Machine learning
Implicit-depth models such as Deep Equilibrium Networks have recently been shown to match or exceed the performance of traditional deep networks while being much more memory efficient.
- 2020-06arXiv 2006.06357Unifying Regularisation Methods for Continual Learning
Frederik Benzing · Machine learning
Continual Learning addresses the challenge of learning a number of different tasks sequentially. The goal of maintaining knowledge of earlier tasks without re-accessing them starkly conflicts with standard SGD training for artificial neural networks.
- 2020-06arXiv 2006.01862Consistent Estimators for Learning to Defer to an Expert
Hussein Mozannar, David Sontag · Machine learning
Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms.
- 2020-05arXiv 2006.04948AI Research Considerations for Human Existential Safety (ARCHES)
Andrew Critch, David Krueger · Society, software & economics
Framed in positive terms, this report examines how technical AI research might be steered in a manner that is more attentive to humanity's long-term prospects for survival as a species.
- 2020-05arXiv 2005.06800Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
Kimin Lee, Younggyo Seo, Seunghyun Lee et al. · Machine learning
Model-based reinforcement learning (RL) enjoys several benefits, such as data-efficiency and planning, by learning a model of the environment's dynamics. However, learning a global model that can generalize across different dynamics is a challenging task.
Cited inStudy: Genesys program prior art - 2020-05arXiv 2005.04118Beyond Accuracy: Behavioral Testing of NLP models with CheckList
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin et al. · Association for Computational Linguistics (ACL), 2020 · Language models & NLP
Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
