NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2022-09arXiv 2209.05364If Influence Functions are the Answer, Then What is the Question?
Juhan Bae, Nathan Ng, Alston Lo et al. · Machine learning
Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters. While influence estimates align well with leave-one-out retraining for linear models, recent works have shown this alignment is often poor in neural networks.
- 2022-09arXiv 2209.04836Git Re-Basin: Merging Models modulo Permutation Symmetries
Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa · Machine learning
The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease.
- 2022-08arXiv 2208.09734A Multi-Head Model for Continual Learning via Out-of-Distribution Replay
Gyuhak Kim, Zixuan Ke, Bing Liu · Machine learning
This paper studies class incremental learning (CIL) of continual learning (CL). Many approaches have been proposed to deal with catastrophic forgetting (CF) in CIL.
- 2022-08arXiv 2208.03306Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
Margaret Li, Suchin Gururangan, Tim Dettmers et al. · Language models & NLP
We present Branch-Train-Merge (BTM), a communication-efficient algorithm for embarrassingly parallel training of large language models (LLMs).
- 2022-08arXiv 2208.02813Towards Understanding Mixture of Experts in Deep Learning
Zixiang Chen, Yihe Deng, Yue Wu et al. · Machine learning
The Mixture-of-Experts (MoE) layer, a sparsely-activated model controlled by a router, has achieved great success in deep learning. However, the understanding of such architecture remains elusive.
- 2022-08arXiv 2208.02814Conformal Risk Control
Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch et al. · Statistics & theory
We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee.
- 2022-07arXiv 2207.10397CodeT: Code Generation with Generated Tests
Bei Chen, Fengji Zhang, Anh Nguyen et al. · Language models & NLP
The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples.
- 2022-07arXiv 2207.05221Language Models (Mostly) Know What They Know
Saurav Kadavath, Tom Conerly, Amanda Askell et al. · Language models & NLP
We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format.
- 2022-06arXiv 2207.00099Measuring Forgetting of Memorized Training Examples
Matthew Jagielski, Om Thakkar, Florian Tramèr et al. · Machine learning
Machine learning models exhibit two seemingly contradictory phenomena: training data memorization, and various forms of forgetting. In memorization, models overfit specific training examples and become susceptible to privacy attacks.
- 2022-06arXiv 2206.13089Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
Christina Baek, Yiding Jiang, Aditi Raghunathan et al. · Machine learning
showed that a model's in-distribution (ID) accuracy has a strong linear correlation with its out-of-distribution (OOD) accuracy on several OOD benchmarks -- a phenomenon they dubbed ''accuracy-on-the-line''.
Cited inInternal notes - 2022-06arXiv 2206.06520Memory-Based Model Editing at Scale
Eric Mitchell, Charles Lin, Antoine Bosselut et al. · AI & agents
Even the largest neural networks make errors, and once-correct predictions can become invalid as the world changes. Model editors make local updates to the behavior of base (pre-trained) models to inject updated knowledge or correct undesirable behaviors.
- 2022-06arXiv 2206.04016SYNERgy between SYNaptic consolidation and Experience Replay for general continual learning
Fahad Sarfraz, Elahe Arani, Bahram Zonooz · Neuroscience & neural computation
Continual learning (CL) in the brain is facilitated by a complex set of mechanisms.
- 2022-06arXiv 2206.02608What do tokens know about their characters and how do they know it?
Ayush Kaushal, Kyle Mahowald · Language models & NLP
Pre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explicit access to the character composition of tokens.
Cited inWhat Language Buys - 2022-05arXiv 2205.14334Teaching Models to Express Their Uncertainty in Words
Stephanie Lin, Jacob Hilton, Owain Evans · Language models & NLP
We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. "90% confidence" or "high confidence").
- 2022-05arXiv 2205.13980The Structure of Online Social Networks Mirror Those in the Offline World
R. I. M. Dunbar, Valerio Arnaboldi, Marco Conti et al. · Social Networks, Volume 43, 2015, Pages 39-47, ISSN 0378-8733 · Society, software & economics
We use data on frequencies of bi-directional posts to define edges (or relationships) in two Facebook datasets and a Twitter dataset and use these to create ego-centric social networks.
- 2022-05arXiv 2205.13323The Effect of Task Ordering in Continual Learning
Samuel J. Bell, Neil D. Lawrence · Machine learning
We investigate the effect of task ordering on continual learning performance. We conduct an extensive series of empirical experiments on synthetic and naturalistic datasets and show that reordering tasks significantly affects the amount of catastrophic forgetting.
- 2022-05arXiv 2205.11822Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jaehun Jung, Lianhui Qin, Sean Welleck et al. · Language models & NLP
Despite their impressive capabilities, large pre-trained language models (LMs) struggle with consistent reasoning; recently, prompting LMs to generate explanations that self-guide the inference has emerged as a promising direction to amend this.
- 2022-05arXiv 2205.10787A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning
Zhi Wang, Chunlin Chen, Daoyi Dong · Machine learning
While reinforcement learning (RL) algorithms are achieving state-of-the-art performance in various challenging tasks, they can easily encounter catastrophic forgetting or interference when faced with lifelong streaming information.
Cited inStudy: Genesys program prior art - 2022-05arXiv 2205.10625Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Denny Zhou, Nathanael Schärli, Le Hou et al. · AI & agents
Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts.
- 2022-05arXiv 2205.09588How catastrophic can catastrophic forgetting be in linear regression?
Itay Evron, Edward Moroshko, Rachel Ward et al. · 35th Annual Conference on Learning Theory (2022) · Machine learning
To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions.
- 2022-05arXiv 2205.10128Neural-Symbolic Models for Logical Queries on Knowledge Graphs
Zhaocheng Zhu, Mikhail Galkin, Zuobai Zhang et al. · AI & agents
Answering complex first-order logic (FOL) queries on knowledge graphs is a fundamental task for multi-hop reasoning. Traditional symbolic methods traverse a complete knowledge graph to extract the answers, which provides good interpretation for each step.
- 2022-04arXiv 2204.14211TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Joel Jang, Seonghyeon Ye, Changho Lee et al. · Language models & NLP
Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment.
- 2022-04arXiv 2204.09179On the Representation Collapse of Sparse Mixture of Experts
Zewen Chi, Li Dong, Shaohan Huang et al. · Language models & NLP
Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations.
- 2022-04arXiv 2204.08396StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma et al. · Machine learning
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference.
- 2022-04arXiv 2204.06507Out-of-Distribution Detection with Deep Nearest Neighbors
Yiyou Sun, Yifei Ming, Xiaojin Zhu et al. · Machine learning
Out-of-distribution (OOD) detection is a critical task for deploying machine learning models in the open world. Distance-based methods have demonstrated promise, where testing samples are detected as OOD if they are relatively far away from in-distribution (ID) data.
- 2022-04arXiv 2204.05624Continual Predictive Learning from Videos
Geng Chen, Wendong Zhang, Han Lu et al. · Vision
Predictive learning ideally builds the world model of physical processes in one or more given environments. Typical setups assume that we can collect data from all environments at all times.
- 2022-04arXiv 2204.04799DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi et al. · Machine learning
Continual learning aims to enable a single model to learn a sequence of tasks without catastrophic forgetting.
- 2022-03arXiv 2203.14465STaR: Bootstrapping Reasoning With Reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu et al. · Machine learning
Generating step-by-step "chain-of-thought" rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering.
- 2022-03arXiv 2203.13381Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
MohammadReza Davari, Nader Asadi, Sudhir Mudur et al. · Machine learning
Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes.
Cited inInternal notes - 2022-03arXiv 2203.11171Self-Consistency Improves Chain of Thought Reasoning in Language Models
Xuezhi Wang, Jason Wei, Dale Schuurmans et al. · Language models & NLP
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting.
- 2022-03arXiv 2203.09450Continual Learning Based on OOD Detection and Task Masking
Gyuhak Kim, Sepideh Esmaeilpour, Changnan Xiao et al. · Vision
Existing continual learning techniques focus on either task incremental learning (TIL) or class incremental learning (CIL) problem, but not both. CIL and TIL differ mainly in that the task-id is provided for each test sample during testing for TIL, but not provided for CIL.
Cited inStudy: Genesys program prior art - 2022-03arXiv 2203.05482Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre et al. · Machine learning
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.
- 2022-03arXiv 2203.02155Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang et al. · Language models & NLP
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user.
- 2022-02arXiv 2202.09368Mixture-of-Experts with Expert Choice Routing
Yanqi Zhou, Tao Lei, Hanxiao Liu et al. · Machine learning
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged.
- 2022-02arXiv 2202.08906ST-MoE: Designing Stable and Transferable Sparse Expert Models
Barret Zoph, Irwan Bello, Sameer Kumar et al. · Language models & NLP
Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models.
- 2022-02arXiv 2202.05262Locating and Editing Factual Associations in GPT
Kevin Meng, David Bau, Alex Andonian et al. · Language models & NLP
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations.
- 2022-02arXiv 2202.03286Red Teaming Language Models with Language Models
Ethan Perez, Saffron Huang, Francis Song et al. · Language models & NLP
Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases.
- 2022-02arXiv 2202.01169Unified Scaling Laws for Routed Language Models
Aidan Clark, Diego de las Casas, Aurelia Guy et al. · Language models & NLP
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input.
- 2022-02arXiv 2202.01197VOS: Learning What You Don't Know by Virtual Outlier Synthesis
Xuefeng Du, Zhaoning Wang, Mu Cai et al. · Machine learning
Out-of-distribution (OOD) detection has received much attention lately due to its importance in the safe deployment of neural networks. One of the key challenges is that models lack supervision signals from unknown data, and as a result, can produce overconfident predictions on OOD data.
- 2022-01arXiv 2201.12604Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
Elahe Arani, Fahad Sarfraz, Bahram Zonooz · Machine learning
Humans excel at continually learning from an ever-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forgetting.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
