NeuroGenesysThe research we build on.
1,514 papers by other researchers that our work cites, from continual learning and verification to language models, agents and hardware. Each entry links to the paper on arXiv and to where we use it. 301 are cited on published pages of this site; the rest in the research notes behind our studies. Our own papers are in the archive.
- 2025-03arXiv 2504.00180Contradiction Detection in RAG Systems: Evaluating LLMs as Context Validators for Improved Information Consistency
Vignesh Gokul, Srikanth Tenneti, Alwarappan Nakkiran · Language models & NLP
Retrieval Augmented Generation (RAG) systems have emerged as a powerful method for enhancing large language models (LLMs) with up-to-date information.
- 2025-03arXiv 2503.13657Why Do Multi-Agent LLM Systems Fail?
Mert Cemri, Melissa Z. Pan, Shuyi Yang et al. · AI & agents
Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal. This gap highlights a critical need for a principled understanding of why MAS fail.
- 2025-03arXiv 2503.05683WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Lukas Thede, Karsten Roth, Matthias Bethge et al. · Language models & NLP
Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weightsStudy: Logical coherence of LLM world modelsStudy: Orchestrated specialists and bridge synthesisStudy: Weight editing to prevent forgetting - 2025-02arXiv 2502.08788Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity
Hangfan Zhang, Zhiyao Cui, Jianhao Chen et al. · Language models & NLP
Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs).
- 2025-02arXiv 2502.05793On Reference (In-)Determinacy in Natural Language Inference
Sihao Chen, Chaitanya Malaviya, Alex Fabrikant et al. · Language models & NLP
, the premise and hypothesis are assumed to refer to the same context when human raters annotate a label.
- 2025-02arXiv 2502.04313Great Models Think Alike and this Undermines AI Oversight
Shashwat Goel, Joschka Struber, Ilze Amanda Auzina et al. · Machine learning
As Language Model (LM) capabilities advance, evaluating and supervising them at scale is getting harder for humans. There is hope that other language models can automate both these tasks, which we refer to as ''AI Oversight''.
- 2025-02arXiv 2502.00674Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
Wenzhe Li, Yong Lin, Mengzhou Xia et al. · Language models & NLP
Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple different Large Language Models (LLMs).
- 2025-01arXiv 2501.14082Communicating Activations Between Language Model Agents
Vignav Ramesh, Kenneth Li · Language models & NLP
Communication between multiple language model (LM) agents has been shown to scale up the reasoning ability of LMs.
- 2024-12arXiv 2412.17009Expert Routing with Synthetic Data for Continual Learning
Yewon Byun, Sanket Vaibhav Mehta, Saurabh Garg et al. · Machine learning
In many real-world settings, regulations and economic incentives permit the sharing of models but not data across institutional boundaries. In such scenarios, practitioners might hope to adapt models to new domains, without losing performance on previous domains (so-called catastrophic forgetting).
- 2024-12arXiv 2412.16100Logical Consistency of Large Language Models in Fact-checking
Bishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat et al. · Language models & NLP
In recent years, large language models (LLMs) have demonstrated significant success in performing varied natural language tasks such as language translation, question-answering, summarizing, fact-checking, etc.
- 2024-12arXiv 2412.06559ProcessBench: Identifying Process Errors in Mathematical Reasoning
Chujie Zheng, Zhenru Zhang, Beichen Zhang et al. · AI & agents
As language models regularly make mistakes when solving math problems, automated identification of errors in the reasoning process becomes increasingly significant for their scalable oversight.
- 2024-11arXiv 2411.04468Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
Adam Fourney, Gagan Bansal, Hussein Mozannar et al. · AI & agents
Modern AI agents, driven by advances in large foundation models, promise to enhance our productivity and transform our lives by augmenting our knowledge and capabilities.
- 2024-09arXiv 2409.07431Synthetic continued pretraining
Zitong Yang, Neil Band, Shuangping Li et al. · Machine learning
Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it.
Cited inCoherence is selected, anchored, then pruned: a literature synthesis on logical coherence in LLM world modelsStudy: Continual learning order and companion dataStudy: Generational competitive continual learningStudy: Logical coherence of LLM world modelsStudy: Orchestrated specialists and bridge synthesis - 2024-07arXiv 2407.06322MagMax: Leveraging Model Merging for Seamless Continual Learning
Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzciński et al. · Machine learning
This paper introduces a continual learning approach named MagMax, which utilizes model merging to enable large pre-trained models to continuously learn from new data without forgetting previously acquired knowledge.
- 2024-07arXiv 2407.04622On scalable oversight with weak LLMs judging strong LLMs
Zachary Kenton, Noah Y. Siegel, János Kramár et al. · Machine learning
Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI.
- 2024-06arXiv 2406.19276VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation
Yixiao Song, Yekyung Kim, Mohit Iyyer · Language models & NLP
, 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia., can plausibly be proven true or false). We address this issue with VERISCORE, a metric for diverse long-form generation tasks that contain both verifiable and unverifiable content.
- 2024-06arXiv 2406.18665RouteLLM: Learning to Route LLMs with Preference Data
Isaac Ong, Amjad Almahairi, Vincent Wu et al. · Machine learning
Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost.
Cited inRoute on shared statistics, never expert confidence: the router architecture for frozen expertsStudy: Hierarchical and T shaped MoEInternal notesStudy: Model self knowledge and confident errorsStudy: Optimal router architectureStudy: Orchestrated specialists and bridge synthesisStudy: Stopping confident wrong expert votes - 2024-06arXiv 2406.06461Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies
Junlin Wang, Siddhartha Jain, Dejiao Zhang et al. · Language models & NLP
A diverse array of reasoning strategies has been proposed to elicit the capabilities of large language models. However, in this paper, we point out that traditional evaluations which focus solely on performance metrics miss a key factor: the increased effectiveness due to additional compute.
- 2024-06arXiv 2406.04692Mixture-of-Agents Enhances Large Language Model Capabilities
Junlin Wang, Jue Wang, Ben Athiwaratkun et al. · Language models & NLP
Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction.
- 2024-06arXiv 2406.01297When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Ryo Kamoi, Yusen Zhang, Nan Zhang et al. · Transactions of the Association for Computational Linguistics (2024) 12: 1417-1440 · Language models & NLP
Self-correction is an approach to improving responses from large language models (LLMs) by refining the responses using LLMs during inference. Prior work has proposed various self-correction frameworks using different sources of feedback, including self-evaluation and external feedback.
- 2024-05arXiv 2405.14768WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Peng Wang, Zexi Li, Ningyu Zhang et al. · Language models & NLP
Large language models (LLMs) need knowledge updates to meet the ever-growing world facts and correct the hallucinated responses, facilitating the methods of lifelong model editing.
Cited inEWC protects what its Fisher sees: a literature synthesis on elastic weight consolidation, scale and coherenceStudy: Consequence consistent updatesStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: LLMs modifying open model weightsStudy: Orchestrated specialists and bridge synthesis - 2024-05arXiv 2405.11157Towards Modular LLMs by Building and Reusing a Library of LoRAs
Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti et al. · Machine learning
The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for new tasks.
- 2024-05arXiv 2405.05904Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Zorik Gekhman, Gal Yona, Roee Aharoni et al. · Language models & NLP
When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training.
Cited inStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errorsTo stop a model forgetting, rehearse what it knew. New context won't do it.Study: Consequence consistent updatesStudy: Continual learning order and companion dataStudy: Fortifying knowledge in continual learningStudy: Generational competitive continual learningStudy: Graded succession by earned authorityStudy: Logical coherence of LLM world modelsStudy: Model self knowledge and confident errorsStudy: Orchestrated specialists and bridge synthesisStudy: Stopping confident wrong expert votes - 2024-04arXiv 2404.16130From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Darren Edge, Ha Trinh, Newman Cheng et al. · Language models & NLP
The use of retrieval-augmented generation (RAG) to retrieve relevant information from an external knowledge source enables large language models (LLMs) to answer questions over private and/or previously unseen document collections.
- 2024-04arXiv 2404.13628Mixture of LoRA Experts
Xun Wu, Shaohan Huang, Furu Wei · Language models & NLP
LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques.
- 2024-03arXiv 2404.00228InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning
Yan-Shuo Liang, Wu-Jun Li · Machine learning
Continual learning requires the model to learn multiple tasks sequentially. In continual learning, the model should possess the ability to maintain its performance on old tasks (stability) and the ability to adapt to new tasks continuously (plasticity).
- 2024-03arXiv 2403.18802Long-form factuality in large language models
Jerry Wei, Chengrun Yang, Xinying Song et al. · Language models & NLP
Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics.
- 2024-03arXiv 2403.12031RouterBench: A Benchmark for Multi-LLM Routing System
Qitian Jason Hu, Jacob Bieker, Xiuyu Li et al. · Machine learning
As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical.
- 2024-03arXiv 2403.08763Simple and Scalable Strategies to Continually Pre-train Large Language Models
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta et al. · Machine learning
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training.
Cited inTo keep what a model knows, control what it is allowed to moveStudy: Elastic weight consolidation deep diveStudy: Fortifying knowledge in continual learningStudy: Hardware guardian core for continual learningStudy: Hierarchical and T shaped MoEStudy: Orchestrated specialists and bridge synthesis - 2024-03arXiv 2403.07816Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma et al. · Language models & NLP
We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge.
Cited inRoute on shared statistics, never expert confidence: the router architecture for frozen expertsStudy: Fortifying knowledge in continual learningStudy: Hierarchical and T shaped MoEStudy: Merging and splitting MoE expertsStudy: Optimal router architectureStudy: Orchestrated specialists and bridge synthesis - 2024-03arXiv 2403.02419Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
Lingjiao Chen, Jared Quincy Davis, Boris Hanin et al. · Machine learning
Many recent state-of-the-art results in language tasks were achieved using compound systems that perform multiple Language Model (LM) calls and aggregate their responses.
- 2024-02arXiv 2402.18272Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?
Qineng Wang, Zihao Wang, Ying Su et al. · Language models & NLP
Recent progress in LLMs discussion suggests that multi-agent discussion improves the reasoning abilities of LLMs. In this work, we reevaluate this claim through systematic experiments, where we propose a novel group discussion framework to enrich the set of discussion mechanisms.
- 2024-02arXiv 2402.09997LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild
Ziyu Zhao, Leilei Gan, Guoyin Wang et al. · AI & agents
Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for fine-tuning large language models (LLM). The modular and plug-and-play nature of LoRA enables the integration of diverse domain-specific LoRAs to enhance the capabilities of LLMs.
- 2024-02arXiv 2402.06782Debating with More Persuasive LLMs Leads to More Truthful Answers
Akbir Khan, John Hughes, Dan Valentine et al. · AI & agents
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts.
- 2024-02arXiv 2402.05859Learning to Route Among Specialized Experts for Zero-Shot Generalization
Mohammed Muqeeth, Haokun Liu, Yufan Liu et al. · Machine learning
Recently, there has been a widespread proliferation of "expert" language models that are specialized to a specific task or domain through parameter-efficient fine-tuning.
- 2024-01arXiv 2401.07453Model Editing at Scale leads to Gradual and Catastrophic Forgetting
Akshat Gupta, Anurag Rao, Gopala Anumanchipalli · Language models & NLP
Editing knowledge in large language models is an attractive capability to have which allows us to correct incorrectly learnt facts during pre-training, as well as update the model with an ever-growing list of new facts.
- 2024-01arXiv 2401.04700Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma et al. · Language models & NLP
Model editing is a technique that edits the large language models (LLMs) with updated knowledge to alleviate hallucinations without resource-intensive retraining.
- 2023-12arXiv 2312.09390Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner et al. · Language models & NLP
Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate whether a model faithfully followed instructions or generated safe outputs.
- 2023-12arXiv 2312.05934Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
Oded Ovadia, Menachem Brief, Moshik Mishaeli et al. · AI & agents
Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains.
- 2023-11arXiv 2311.17311Universal Self-Consistency for Large Language Model Generation
Xinyun Chen, Renat Aksitov, Uri Alon et al. · Language models & NLP
Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled from large language models (LLMs).
Built from every arXiv paper cited in our papers, posts, topic hubs, reports and research notes, with titles, authors, dates and abstracts from the arXiv API (updated 2026-11-12). Summaries are the opening sentences of each paper's own abstract, not our assessment; a citation means we used or checked the work, not that we endorse it. Fields follow each paper's primary arXiv category. Papers cited only outside arXiv (journals, proceedings) are not listed yet. Entries are generated from our citations, so a mistyped ID in a note would list the wrong paper; we remove those as we find them. Report a wrong entry.
