NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2022-01arXiv 2201.05596DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Samyam Rajbhandari, Conglong Li, Zhewei Yao et al. · Machine learning
As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model.
- 2022-01arXiv 2201.04122In Defense of the Unitary Scalarization for Deep Multi-Task Learning
Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov et al. · Machine learning
Recent multi-task learning research argues against unitary scalarization, where training simply minimizes the sum of the task losses. Several ad-hoc multi-task optimization algorithms have instead been proposed, inspired by various hypotheses about what makes multi-task settings difficult.
- 2022-01arXiv 2201.04234Leveraging Unlabeled Data to Predict Out-of-Distribution Performance
Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton et al. · Machine learning
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops.
Cited inInternal notes - 2021-12arXiv 2112.08654Learning to Prompt for Continual Learning
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee et al. · Machine learning
The mainstream paradigm behind continual learning has been to adapt the model parameters to non-stationary data distributions, where catastrophic forgetting is the central challenge.
- 2021-12arXiv 2112.00861A General Language Assistant as a Laboratory for Alignment
Amanda Askell, Yuntao Bai, Anna Chen et al. · Language models & NLP
Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless.
- 2021-11arXiv 2111.09832Merging Models with Fisher-Weighted Averaging
Michael Matena, Colin Raffel · Machine learning
Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities.
- 2021-11arXiv 2111.05498Attention Approximates Sparse Distributed Memory
Trenton Bricken, Cengiz Pehlevan · 35th Conference on Neural Information Processing Systems (NeurIPS 2021) · Machine learning
While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well.
- 2021-10arXiv 2110.14168Training Verifiers to Solve Math Word Problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian et al. · Machine learning
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning.5K high quality linguistically diverse grade school math word problems.
- 2021-10arXiv 2110.13715ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs
Zhanqiu Zhang, Jie Wang, Jiajun Chen et al. · AI & agents
Query embedding (QE) -- which aims to embed entities and first-order logical (FOL) queries in low-dimensional spaces -- has shown great power in multi-hop reasoning over knowledge graphs.
- 2021-10arXiv 2110.14048Conflict-Averse Gradient Descent for Multi-task Learning
Bo Liu, Xingchao Liu, Xiaojie Jin et al. · Machine learning
The goal of multi-task learning is to enable more efficient learning than single task learning by sharing model structures for a diverse set of tasks. A standard multi-task learning objective is to minimize the average loss across all tasks.
- 2021-10arXiv 2110.11309Fast Model Editing at Scale
Eric Mitchell, Charles Lin, Antoine Bosselut et al. · Machine learning
While large pre-trained models have enabled impressive results on a variety of downstream tasks, the largest existing models still make errors, and even accurate predictions may become outdated over time.
- 2021-10arXiv 2110.06296The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
Rahim Entezari, Hanie Sedghi, Olga Saukh et al. · Machine learning
In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them.
- 2021-10arXiv 2110.03215Towards Continual Knowledge Learning of Language Models
Joel Jang, Seonghyeon Ye, Sohee Yang et al. · Language models & NLP
Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue.
- 2021-10arXiv 2110.01786MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu et al. · Language models & NLP
Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear.
- 2021-10arXiv 2110.01052Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès et al. · Machine learning
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting.
- 2021-10arXiv 2110.00175DualNet: Continual Learning, Fast and Slow
Quang Pham, Chenghao Liu, Steven Hoi · Machine learning
According to Complementary Learning Systems (CLS) theory~\citep{mcclelland1995there} in neuroscience, humans do effective \emph{continual learning} through two complementary systems: a fast learning system centered on the hippocampus for rapid learning of the specifics and individual experiences, and a slow…
- 2021-09arXiv 2109.14723BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
Nora Kassner, Oyvind Tafjord, Hinrich Schütze et al. · Language models & NLP
Although pretrained language models (PTLMs) contain significant amounts of world knowledge, they can still produce inconsistent answers to questions when probed, even after specialized training.
- 2021-09arXiv 2109.05052Entity-Based Knowledge Conflicts in Question Answering
Shayne Longpre, Kartik Perisetla, Anthony Chen et al. · Language models & NLP
Knowledge-dependent tasks typically use two sources of knowledge: parametric, learned at training time, and contextual, given as a passage at inference time.
- 2021-09arXiv 2109.07958TruthfulQA: Measuring How Models Mimic Human Falsehoods
Stephanie Lin, Jacob Hilton, Owain Evans · Language models & NLP
We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics.
- 2021-09arXiv 2109.01903Robust fine-tuning of zero-shot models
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim et al. · Vision
, without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts.
- 2021-07arXiv 2107.01057Backward-Compatible Prediction Updates: A Probabilistic Approach
Frederik Träuble, Julius von Kügelgen, Matthäus Kleindessner et al. · Machine learning
When machine learning systems meet real world applications, accuracy is only one of several requirements. In this paper, we assay a complementary perspective originating from the increasing availability of pre-trained and regularly improving state-of-the-art models.
- 2021-06arXiv 2106.15110Time-Aware Language Models as Temporal Knowledge Bases
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos et al. · Transactions of the Association for Computational Linguistics 2022; 10 257-273 · Language models & NLP
Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for.
- 2021-06arXiv 2106.14342Stabilizing Equilibrium Models by Jacobian Regularization
Shaojie Bai, Vladlen Koltun, J. Zico Kolter · Machine learning
Deep equilibrium networks (DEQs) are a new class of models that eschews traditional depth in favor of finding the fixed point of a single nonlinear layer. These models have been shown to achieve performance competitive with the state-of-the-art deep networks while using significantly less memory.
- 2021-06arXiv 2106.14806Laplace Redux -- Effortless Bayesian Deep Learning
Erik Daxberger, Agustinus Kristiadi, Alexander Immer et al. · Machine learning
Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection.
- 2021-06arXiv 2106.08769Knowledge-Adaptation Priors
Mohammad Emtiyaz Khan, Siddharth Swaroop · Machine learning
Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch.
- 2021-06arXiv 2106.09022A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
Jie Ren, Stanislav Fort, Jeremiah Liu et al. · Machine learning
Mahalanobis distance (MD) is a simple and popular post-processing method for detecting out-of-distribution (OOD) inputs in neural networks.
- 2021-06arXiv 2106.08085Natural continual learning: success is a journey, not (just) a destination
Ta-Chu Kao, Kristopher T. Jensen, Gido M. van de Ven et al. · Machine learning
Biological agents are known to learn many different tasks over the course of their lives, and to be able to revisit previous tasks and behaviors with little to no loss in performance.
- 2021-06arXiv 2106.05945Does Knowledge Distillation Really Work?
Samuel Stanton, Pavel Izmailov, Polina Kirichenko et al. · Machine learning
Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks.
- 2021-06arXiv 2106.04426Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam et al. · Machine learning
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence.
- 2021-05arXiv 2105.10919Continual World: A Robotic Benchmark For Continual Reinforcement Learning
Maciej Wołczyk, Michał Zając, Razvan Pascanu et al. · Machine learning
Continual learning (CL) -- the ability to continuously learn, building on previously acquired knowledge -- is a natural requirement for long-lived autonomous reinforcement learning (RL) agents.
Cited inStudy: Genesys program prior art - 2021-05arXiv 2105.03048Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates
Yuqing Xie, Yi-an Lai, Yuanjun Xiong et al. · Language models & NLP
Behavior of deep neural networks can be inconsistent between different versions. Regressions during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain.
- 2021-04arXiv 2104.08696Knowledge Neurons in Pretrained Transformers
Damai Dai, Li Dong, Yaru Hao et al. · Language models & NLP
Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.
- 2021-04arXiv 2104.03344OVANet: One-vs-All Network for Universal Domain Adaptation
Kuniaki Saito, Kate Saenko · Vision
Universal Domain Adaptation (UNDA) aims to handle both domain-shift and category-shift between two datasets, where the main challenge is to transfer knowledge while rejecting unknown classes which are absent in the labeled source data but present in the unlabeled target data.
- 2021-03arXiv 2103.16716BASE Layers: Simplifying Training of Large, Sparse Models
Mike Lewis, Shruti Bhosale, Tim Dettmers et al. · Language models & NLP
We introduce a new balanced assignment of experts (BASE) layer for large language models that greatly simplifies existing high capacity sparse layers.
- 2021-03arXiv 2103.09762Gradient Projection Memory for Continual Learning
Gobinda Saha, Isha Garg, Kaushik Roy · International Conference on Learning Representations (ICLR), 2021 · Machine learning
The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory.
- 2021-03arXiv 2103.07113Training Networks in Null Space of Feature Covariance for Continual Learning
Shipeng Wang, Xiaorong Li, Jian Sun et al. · Machine learning
In the setting of continual learning, a network is trained on a sequence of tasks, and suffers from catastrophic forgetting.
- 2021-03arXiv 2103.00953Adversarial Reciprocal Points Learning for Open Set Recognition
Guangyao Chen, Peixi Peng, Xiangqian Wang et al. · IEEE Transactions on Pattern Analysis and Machine Intelligence 2021 · Vision
The key challenge of OSR is how to reduce the empirical classification risk on the labeled known data and the open space risk on the potential unknown data simultaneously.
- 2021-02arXiv 2102.13202Online Multi-Armed Bandits with Adaptive Inference
Maria Dimakopoulou, Zhimei Ren, Zhengyuan Zhou · Machine learning
During online decision making in Multi-Armed Bandits (MAB), one needs to conduct inference on the true mean reward of each arm based on data collected so far at each step.
- 2021-02arXiv 2102.11582Deep Deterministic Uncertainty: A Simple Baseline
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort et al. · Machine learning
Reliable uncertainty from deterministic single-forward pass models is sought after because conventional methods of uncertainty quantification are computationally expensive.
- 2021-02arXiv 2102.11174Linear Transformers Are Secretly Fast Weight Programmers
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber · Machine learning
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of…
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
