To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning.
Abstract
Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's queryspecific prior capabilities and continuous onpolicy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL. 1 Introduction Hallucination remains a long-standing and notoriously difficult challenge in LLMs. It is challenging to ensure the factual grounding of generated output. (Bang et al., 2025; Kalai et al., 2025) A prevailing approach to mitigating hallucinations is providing sufficient context to ground the answers. Progressing from early Retrieval-Augmented Generation methods, (Agrawal et al., 2024; Béchard and Ayala, 2024) recent advancements have introduced more advanced search agents. (Li et al., 2025b,a) By leveraging outcome-reward RL, these agents learn to interact with search environments iteratively to arrive at ground-truth answers. However, this training paradigm harbors a critical vulnerability: current RL objectives predom- inantly incentivize models to output the correct answer, but lack a mechanism to reward abstention. (Jin et al., 2025) Consequently, when an agent encounters queries beyond its capability boundaries or fails to retrieve sufficient evidence, it tends to fabricate an answer rather than safely abstain. This "over-answering" phenomenon could exacerbate hallucinat