NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2024-06arXiv 2406.14026Demystifying Language Model Forgetting with Low-rank Example Associations
Xisen Jin, Xiang Ren · Machine learning
Large language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks.
- 2024-06arXiv 2406.03689Evaluating the World Model Implicit in a Generative Model
Keyon Vafa, Justin Y. Chen, Ashesh Rambachan et al. · Language models & NLP
Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is governed by a deterministic finite automaton.
- 2024-06arXiv 2406.01297When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Ryo Kamoi, Yusen Zhang, Nan Zhang et al. · Transactions of the Association for Computational Linguistics (2024) 12: 1417-1440 · Language models & NLP
Self-correction is an approach to improving responses from large language models (LLMs) by refining the responses using LLMs during inference. Prior work has proposed various self-correction frameworks using different sources of feedback, including self-evaluation and external feedback.
- 2024-05arXiv 2405.21030Standards for Belief Representations in LLMs
Daniel A. Herrmann, Benjamin A. Levinstein · Minds and Machines 35:5 (2025) 1-25 · AI & agents
As large language models (LLMs) continue to demonstrate remarkable abilities across various domains, computer scientists are developing methods to understand their cognitive processes, particularly concerning how (and if) LLMs internally represent their beliefs about the world.
- 2024-05arXiv 2405.11613Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts
Baolong Bi, Shenghua Liu, Lingrui Mei et al. · Language models & NLP
The knowledge within large language models (LLMs) may become outdated quickly. While in-context editing (ICE) is currently the most effective method for knowledge editing (KE), it is constrained by the black-box modeling of LLMs and thus lacks interpretability.
- 2024-05arXiv 2405.05904Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Zorik Gekhman, Gal Yona, Roee Aharoni et al. · Language models & NLP
When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training.
Cited inStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errorsTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learningStudy: Graded succession by earned authorityStudy: Logical coherence of LLM world modelsStudy: Model self knowledge and confident errorsStudy: Orchestrated specialists and bridge synthesisStudy: Stopping confident wrong expert votes - 2024-04arXiv 2404.16767REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan et al. · Machine learning
While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications, including the fine-tuning of generative models.
- 2024-04arXiv 2404.01413Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey et al. · Machine learning
The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs?
- 2024-03arXiv 2403.18802Long-form factuality in large language models
Jerry Wei, Chengrun Yang, Xinying Song et al. · Language models & NLP
Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics.
- 2024-03arXiv 2403.08319Knowledge Conflicts for LLMs: A Survey
Rongwu Xu, Zehan Qi, Zhijiang Guo et al. · Language models & NLP
This survey provides an in-depth analysis of knowledge conflicts for large language models (LLMs), highlighting the complex challenges they encounter when blending contextual and parametric knowledge.
- 2024-02arXiv 2402.10978Language Models with Conformal Factuality Guarantees
Christopher Mohri, Tatsunori Hashimoto · Machine learning
Guaranteeing the correctness and factuality of language model (LM) outputs is a major open problem. In this work, we propose conformal factuality, a framework that can ensure high probability correctness guarantees for LMs by connecting language modeling and conformal prediction.
- 2024-02arXiv 2402.06782Debating with More Persuasive LLMs Leads to More Truthful Answers
Akbir Khan, John Hughes, Dan Valentine et al. · AI & agents
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts.
- 2024-02arXiv 2402.04333LESS: Selecting Influential Data for Targeted Instruction Tuning
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan et al. · Language models & NLP
Instruction tuning has unlocked powerful capabilities in large language models (LLMs), effectively using combined datasets to develop generalpurpose chatbots., reasoning).
- 2024-02arXiv 2402.01865What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement
Xisen Jin, Xiang Ren · Machine learning
Language models deployed in the wild make errors. However, simply updating the model with the corrected error instances causes catastrophic forgetting -- the updated model makes errors on instances learned during the instruction tuning or upstream training phase.
Cited inA gradient kernel flags contradictions but does not predict forgetting, and steering rehearsal by it backfiresCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Continual learning order and companion dataStudy: Decision signatures for conflict detectionStudy: Logical coherence of LLM world modelsStudy: Seven continual learning methods tested - 2024-01arXiv 2401.16646Incoherent Probability Judgments in Large Language Models
Jian-Qiao Zhu, Thomas L. Griffiths · CogSci 2024 · Language models & NLP
Autoregressive Large Language Models (LLMs) trained for next-word prediction have demonstrated remarkable proficiency at producing coherent text. But are they equally adept at forming coherent probability judgments?
- 2024-01arXiv 2401.14885Neuromorphic quadratic programming for efficient and scalable model predictive control
Ashish Rao Mangalore, Gabriel Andres Fonseca Guerra, Sumedh R. Risbud et al. · Neuroscience & neural computation
Applications in robotics or other size-, weight- and power-constrained autonomous systems at the edge often require real-time and low-energy solutions to large optimization problems.
- 2024-01arXiv 2401.10020Self-Rewarding Language Models
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho et al. · Language models & NLP
We posit that to achieve superhuman agents, future models require superhuman feedback in order to provide an adequate training signal.
- 2024-01arXiv 2401.08574Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen et al. · Language models & NLP
While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world.
- 2024-01arXiv 2401.00757LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
Yuxuan Wan, Wenxuan Wang, Yiliu Yang et al. · Society, software & economics
We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) such as ChatGPT and GPT-4.
- 2023-12arXiv 2312.10029Challenges with unsupervised LLM knowledge discovery
Sebastian Farquhar, Vikrant Varma, Zachary Kenton et al. · Machine learning
We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent.
- 2023-11arXiv 2312.08748All-to-all reconfigurability with sparse and higher-order Ising machines
Srijan Nikhar, Sidharth Kannan, Navid Anjum Aadit et al. · Nature Communications (2024) · Other
Domain-specific hardware to solve computationally hard optimization problems has generated tremendous excitement. Here, we evaluate probabilistic bit (p-bit) based Ising Machines (IM) on the 3-regular 3-Exclusive OR Satisfiability (3R3X), as a representative hard optimization problem.
- 2023-11arXiv 2311.09603Self-Contradictory Reasoning Evaluation and Detection
Ziyi Liu, Soumya Sanyal, Isabelle Lee et al. · Language models & NLP
In a plethora of recent work, large language models (LLMs) demonstrated impressive reasoning ability, but many proposed downstream reasoning tasks only focus on final answers.
- 2023-11arXiv 2311.08106Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models
Yujin Kim, Jaehong Yoon, Seonghyeon Ye et al. · Language models & NLP
The dynamic nature of knowledge in an ever-changing world presents challenges for language models trained on static data; the model in the real world often requires not only acquiring new knowledge but also overwriting outdated information into updated ones.
- 2023-10arXiv 2310.15164LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers
Theo X. Olausson, Alex Gu, Benjamin Lipkin et al. · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5153-5176, Singapore. Association for Computational Linguistics · Language models & NLP
, deductively inferring the truth value of a conclusion from a set of premises, is an important task for artificial intelligence with wide potential impacts on science, mathematics, and society.
- 2023-10arXiv 2310.13548Towards Understanding Sycophancy in Language Models
Mrinank Sharma, Meg Tong, Tomasz Korbak et al. · Language models & NLP
Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy.
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errorsStudy: Logical coherence of LLM world modelsStudy: Model self knowledge and confident errors - 2023-10arXiv 2310.08540Do pretrained Transformers Learn In-Context by Gradient Descent?
Lingfeng Shen, Aayush Mishra, Daniel Khashabi · Language models & NLP
The emergence of In-Context Learning (ICL) in LLMs remains a remarkable phenomenon that is partially understood. To explain ICL, recent studies have created theoretical connections to Gradient Descent (GD).
- 2023-10arXiv 2310.06824The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
Samuel Marks, Max Tegmark · AI & agents
Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations.
- 2023-10arXiv 2310.02984Scaling Laws for Associative Memories
Vivien Cabannes, Elvis Dohmatob, Alberto Bietti · Machine learning
Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models.
- 2023-10arXiv 2310.01798Large Language Models Cannot Self-Correct Reasoning Yet
Jie Huang, Xinyun Chen, Swaroop Mishra et al. · Language models & NLP
Large Language Models (LLMs) have emerged as a groundbreaking technology with their unparalleled text generation capabilities across various applications. Nevertheless, concerns persist regarding the accuracy and appropriateness of their generated content.
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errorsStudy: Civilization as a superintelligence modelStudy: Cross generational teaching and examinationStudy: Logical coherence of LLM world modelsStudy: Model self knowledge and confident errorsStudy: Orchestrated specialists and bridge synthesis - 2023-10arXiv 2310.02207Language Models Represent Space and Time
Wes Gurnee, Max Tegmark · Machine learning
The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world.
- 2023-10arXiv 2310.01405Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou, Long Phan, Sarah Chen et al. · Machine learning
In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience.
- 2023-09arXiv 2309.12288The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
Lukas Berglund, Meg Tong, Max Kaufmann et al. · Language models & NLP
We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A".
- 2023-09arXiv 2309.11495Chain-of-Verification Reduces Hallucination in Large Language Models
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu et al. · Language models & NLP
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models. We study the ability of language models to deliberate on the responses they give in order to correct their mistakes.
- 2023-09arXiv 2309.00941Emergent Linear Representations in World Models of Self-Supervised Sequence Models
Neel Nanda, Andrew Lee, Martin Wattenberg · Machine learning
How do sequence models represent their decision-making process?, 2023). In this work, we provide evidence of a closely related linear representation of the board.
- 2023-08arXiv 2308.03296Studying Large Language Model Generalization with Influence Functions
Roger Grosse, Juhan Bae, Cem Anil et al. · Machine learning
When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior?
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Hardware guardian core for continual learningStudy: Logical coherence of LLM world models - 2023-07arXiv 2307.12976Evaluating the Ripple Effects of Knowledge Editing in Language Models
Roi Cohen, Eden Biran, Ori Yoran et al. · Language models & NLP
Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations.
Cited inA gradient kernel flags contradictions but does not predict forgetting, and steering rehearsal by it backfiresCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceIterated self-closure settles fast, but only evidence that overrides self-belief makes the settled state trueTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Decision signatures for conflict detectionStudy: Elastic weight consolidation deep diveStudy: Graded succession by earned authorityStudy: Hardware guardian core for continual learningStudy: Logical coherence of LLM world modelsStudy: Orchestrated specialists and bridge synthesis - 2023-06arXiv 2307.00175Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks
B. A. Levinstein, Daniel A. Herrmann · Language models & NLP
We consider the questions of whether or not large language models (LLMs) have beliefs, and, if they do, how we might measure them. First, we evaluate two existing approaches, one due to Azaria and Mitchell (2023) and the other to Burns et al.
- 2023-06arXiv 2306.09983Evaluating Superhuman Models with Consistency Checks
Lukas Fluri, Daniel Paleka, Florian Tramèr · Machine learning
If machine learning models were to achieve superhuman abilities at various reasoning or decision-making tasks, how would we go about evaluating such models, given that humans would necessarily be poor proxies for ground truth?
- 2023-06arXiv 2306.09306Propagating Knowledge Updates to LMs Through Distillation
Shankar Padmanabhan, Yasumasa Onoe, Michael J. Q. Zhang et al. · Language models & NLP
Modern language models have the capacity to store and use immense amounts of knowledge about real-world entities, but it remains unclear how to update such knowledge stored in model parameters.
- 2023-06arXiv 2306.03341Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Kenneth Li, Oam Patel, Fernanda Viégas et al. · Machine learning
We introduce Inference-Time Intervention (ITI), a technique designed to enhance the "truthfulness" of large language models (LLMs). ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads.
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
