Raw capture — raw/the-reasoning-trap.md

url: https://arxiv.org/html/2510.22977v2 date_fetched: 2026-07-05 backfilled: true

The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination

Abstract

Enhancing the reasoning capabilities of Large Language Models (LLMs) is a key strategy for building Agents that “think then act.” However, recent observations, like OpenAI’s o3, suggest a paradox: stronger reasoning often coincides with increased hallucination, yet no prior work has systematically examined whether reasoning enhancement itself causes tool hallucination. To address this gap, we pose the central question: Does strengthening reasoning increase tool hallucination of LLM Agents? We address this gap by introducing SimpleToolHalluBench, a diagnostic benchmark measuring tool hallucination. Through controlled experiments, we show that across RL, distillation, and toggleable reasoning modes, gains in task performance are consistently accompanied by higher tool hallucination rates. This effect is training method-agnostic and transcends simple overfitting, as training even on non-tool-related tasks (e.g., mathematics) still amplifies tool hallucination. Controlled ablations further reveal that the reasoning itself, rather than RL training in general, is most closely associated with the hallucination increase. We evaluate mitigation strategies including Prompt Engineering and Direct Preference Optimization (DPO), revealing a fundamental reliability–capability trade-off: reducing hallucination unavoidably degrades utility. Our findings demonstrate that under current reasoning enhancement methods, improved reasoning is systematically associated with increased tool hallucination, highlighting the need for training objectives that jointly optimize capability and reliability. Our data and codes are available at https://github.com/albert-y1n/Reasoning_Trap.

The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination

Chenlong Yin1, Zeyang Sha2,†, Shiwen Cui3, Changhua Meng3, Zechao Li2 1The Pennsylvania State University 2Nanjing University of Science and Technology 3Independent Researcher

2

2footnotetext: Corresponding author.

1 Introduction

The evolution of Large Language Models (LLMs) from text generators into Agents that interleave internal deliberation with external tool calls has marked a significant step towards accomplishing diverse real-world tasks (Wei et al., 2022; Trivedi et al., 2023; Yao et al., 2023; Schick et al., 2024; Sha et al., 2025a). This agentic shift is largely propelled by advanced reinforcement learning techniques, which have moved beyond merely enriching knowledge to enhancing core capabilities like reasoning and tool use. Collectively, state-of-the-art reinforcement learning algorithms are gradually enabling models to acquire more advanced intelligence.

However, while substantial evidence shows that reasoning abilities enhanced by reinforcement learning can increase hallucinations in model responses (OpenAI, 2025; Chowdhury et al., 2025), its influence on models’ hallucination tendencies during tool use remains largely unexplored. We term this phenomenon tool hallucination, which occurs when models either fabricate non-existent tools or misappropriate available but irrelevant tools, leading to unfounded claims compromising agent reliability (Patil et al., 2024). These hallucinations pose significant risks: they can directly impact real-world systems, generate misleading outputs that are difficult to verify, and critically undermine user trust by producing seemingly plausible but fundamentally flawed tool-based responses. To systematically investigate this emerging challenge, our work is guided by three central research questions:

Our investigation, summarized in Figure 1, systematically addresses these questions. To answer RQ1, we propose SimpleToolHalluBench, a lightweight diagnostic that isolates tool fidelity via two settings: No-Tool-Available task and Distractor-Tool task. Using this diagnostic, we find that reasoning enhancement, whether via RL, distillation, or toggleable modes, is consistently accompanied by increased hallucination, even when training on non-tool tasks like mathematics. To address RQ2, we perform a mechanistic analysis of the model’s dynamics. Layer-wise representation similarity reveals that while in-distribution reasoning pathways remain stable, tool-related representations undergo substantial drift in early and middle layers. Activation probes further show pronounced divergence in deep residual streams, where correct and hallucinated responses become most linearly separable. To address RQ3, we evaluate mitigation strategies and find that prompt-based instructions yield only marginal gains, while Direct Preference Optimization (DPO) meaningfully reduces hallucination at the cost of a substantial utility drop, underscoring the difficulty of reconciling reliability and capability under current training paradigms.

In summary, our contributions are: (1) Introducing SimpleToolHalluBench, a lightweight diagnostic benchmark for measuring tool hallucination under controlled conditions. (2) Providing the first experimental and mechanistic evidence that reasoning-focused RL inherently amplifies tool hallucination across different training methods and model families. (3) Demonstrating a fundamental reliability-capability trade-off, showing that current mitigation strategies reduce hallucination at the direct expense of reasoning performance.

2 Related Work

LLMs as Tool-Using Agents. Chain-of-Thought (CoT) prompting unlocked multi-step reasoning by guiding models to “think step by step”(Wei et al., 2022). Building on this, systems interleave internal reasoning with calls to external knowledge sources for knowledge-intensive tasks (Trivedi et al., 2023). ReAct (Yao et al., 2023) unifies reasoning and acting via interleaved thought traces and tool actions, creating a reason–act–observe loop for iterative planning and error correction. Complementarily, Toolformer (Schick et al., 2024) demonstrates self-supervised tool learning, where LLMs learn when and how to call APIs. Collectively, these works establish LLM agents that couple structured reasoning with purposeful tool use.

Reinforcement Learning for Reasoning. Early PPO-style approaches incorporate process-aware supervision—e.g., token/step-level shaping that rewards correct intermediate reasoning (Stiennon et al., 2020). By contrast, newer algorithms such as GRPO optimize outcome-level signals by comparing groups of complete responses, yielding stronger credit assignment from result quality alone (Shao et al., 2024; Guo et al., 2025). This outcome-centric recipe increasingly powers agentic reasoning, including agents that reason with search engines (Jin et al., 2025; Song et al., 2025; Zheng et al., 2025; Chen et al., 2025) and frameworks that optimize tool-calling policies (Li et al., 2025; Qian et al., 2025; Wang et al., 2025a, b).

Hallucination in LLMs. Despite rapid progress, LLMs can hallucinate—producing nonsensical or unfaithful content (Zhang et al., 2025), raising concerns about honesty and reliability (Li et al., 2024; Gao et al., 2024; Sha et al., 2025b). With tool use, a specialized failure mode emerges: tool hallucination, including fabricating tools, invoking tools with incorrect parameters, or misreading tool outputs. Such errors can cascade through multi-step plans, undermining reliability even when intermediate reasoning appears coherent. Diagnostic benchmarks such as ToolBeHonest (Zhang et al., 2024a) quantify these errors; early mitigation strategies aim to reduce their frequency without degrading capability (Xu et al., 2024).

3 SimpleToolHalluBench: A Benchmark for Tool Hallucination

Current benchmarks primarily focus on whether models can accurately invoke tools in complete environments, but a fundamental question remains insufficiently studied: can agents reliably abstain from tool use when no appropriate tools are available?

This abstention capability is critical for real-world deployment, where agents must distinguish between solvable and unsolvable queries given their available tools. Inappropriate tool invocation, such as fabricating non-existent tools or misusing available but irrelevant ones, leads to unfounded tool-based claims that compromise the reliability of agent interactions.

We observe that many agents fail at this fundamental abstention behavior even in simple scenarios. To diagnose these fundamental failures, we introduce SimpleToolHalluBench, a lightweight benchmark for measuring tool hallucination. Our benchmark is designed to systematically probe an agent’s tendency to hallucinate tool invocations when it should instead abstain from action.

3.1 Benchmark Design

We identify two fundamental scenarios where tool hallucination commonly occurs. Our benchmark, SimpleToolHalluBench, systematically probes these failure modes through controlled experimental conditions.

No-Tool-Available Task (NTA).

This task tests whether agents can recognize when they lack the necessary tools to fulfill a request. The system prompt provides no tools, but the user query explicitly requires external tool invocation to be answered correctly (e.g., “What is the current time in Park Forest Village?”). We measure whether agents instead hallucinate the existence and output of non-existent tools (e.g., fabricating a get_current_time function and its output).

Distractor-Tool Task (DT).

This task evaluates whether agents can recognize that distractor tools cannot fulfill the user’s query and acknowledge the absence of the necessary tool. The system prompt includes a distractor tool that is irrelevant to the user query (e.g., a ’calculator’ tool is provided for a weather query). The query itself requires a different, unprovided tool. We evaluate whether the agent incorrectly attempts to use the distractor tool or hallucinates a more appropriate one.

We construct SimpleToolHalluBench by selecting 296 tools from AgentSafetyBench Zhang et al. (2024b) and generating corresponding user queries using ChatGPT-4o. Crucially, we ensure that each query can only be correctly answered by invoking its specific corresponding tool, meaning the queries cannot be resolved through internal model knowledge or any other available tools. This design guarantee means that in both NTA and DT settings, where the required tool is unavailable, the queries are absolutely impossible to complete correctly.

We calculate the hallucination rate for each task independently. These rates are defined as the fraction of responses flagged by the LLM-as-judge:

| (1) |

where and are the counts of hallucinated responses in their respective tasks, and and are the total number of samples in each task set. More implementation details for SimpleToolHalluBench are provided in Appendix A.

4 Tool Hallucination in Reasoning RL

Having established SimpleToolHalluBench as our diagnostic tool for measuring tool hallucination, we now systematically investigate whether and how reasoning enhancement leads to this failure mode. We conduct four sequential experiments to isolate the root cause. First, we test the most direct hypothesis: does tool-specific Reasoning RL, i.e., training agents explicitly on tool-reasoning tasks, increase hallucination rates? Finding a strong positive correlation, we face a critical ambiguity: is this merely overfitting to tool-use patterns, or does reasoning enhancement itself drive hallucination? Our second experiment addresses this by applying Reasoning RL to pure mathematical problems with no tool involvement whatsoever, yet still observing increased tool hallucination downstream. Third, we examine whether this phenomenon extends beyond RL-based methods by evaluating distillation and toggleable reasoning modes. Finally, we perform controlled ablations to further narrow the locus of the effect, showing that reasoning itself, rather than RL training in general or broader instruction-following degradation, is most closely associated with the hallucination increase. Together, these experiments reveal that reasoning enhancement, regardless of its specific instantiation, is systematically associated with amplified tool hallucination.

4.1 The Side-Effects of Tool-Specific Reasoning RL

Our first experiment investigates whether direct reinforcement learning on agentic tasks exacerbates hallucination. We replicate ReCall (Chen et al., 2025), a state-of-the-art GRPO-style agentic reasoning framework, using Qwen2.5-7B-Instruct as our base LLM. Following the original methodology, we apply GRPO on the SynTool training split, saving checkpoints every 100 steps. Implementation details are detailed in Appendix D.

Each checkpoint undergoes dual evaluation: we measure task utility via the SynTool validation reward and assess hallucination tendency using SimpleToolHalluBench. The results reveal a clear and concerning trade-off. As illustrated in Figure 2(b), the SynTool validation reward steadily improves, confirming the RL effectively optimizes the target objective. However, Figure 2(a) shows a simultaneous, substantial increase in hallucination rates on both No-Tool-Available (NTA) and Distractor-Tool (DT) tasks. This establishes a strong correlation: agents explicitly rewarded for generating tool-use reasoning chains become over-eager to apply this behavior, even in contexts where tools are missing, irrelevant, or should be abstained from.

4.2 Non-Agentic Reasoning RL Can Also Be a Driver of Tool Hallucination

While these results clearly demonstrate that tool-specific RL amplifies hallucination, they leave a critical question unanswered: is this phenomenon merely a consequence of overfitting to tool-use patterns, or does it reflect a more fundamental property of reasoning enhancement itself? To disentangle these possibilities, we designed a further experiment that removes tool-related training data.

We conduct experiments on the GSM8K dataset (Cobbe et al., 2021), a collection of math problems requiring step-by-step logical reasoning but entirely unrelated to external tools. We fine-tune the model using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), following standard practices for reasoning-focused RL. As in the previous experiment, we save checkpoints at regular intervals and evaluate each on both the GSM8K validation set and on SimpleToolHalluBench. The details of GRPO are provided in Appendix C.1.

The training results demonstrate effective learning: Figure 3(b) shows that the model’s accuracy on GSM8K validation steadily improves, reflecting successful acquisition of mathematical reasoning skills. However, despite the absence of any tool-related supervision, we again observe a consistent rise in hallucination rates on both No-Tool-Available Task (NTA) and Distractor-Tool Task (DT) as training progresses (Figure 3(a)).

This finding highlights a crucial insight. The increase of tool hallucination cannot be fully attributed to overfitting on tool-use data. Instead, the reinforcement of confident, chain-of-thought style reasoning appears to instill a general tendency to fill in the gaps with plausible but unsupported content. When placed in tool-use scenarios, this behavior naturally manifests as tool hallucination. Thus, reasoning enhancement itself, not just its tool-specific application, emerges as a key factor associated with increased hallucination.

Taken together, our two experiments demonstrate that while tool-specific RL accelerates hallucination in obvious ways, the root cause lies deeper: reinforcing reasoning chains inherently biases models toward generating confident but unfounded outputs, which surface as tool hallucination when external tools are involved.

4.3 Generalizing the Impact of Reasoning on Tool Hallucination

Having established that both tool-specific and non-agentic reasoning RL increase tool hallucination, we next investigate whether this phenomenon extends beyond RL-based training methods. If reasoning enhancement itself, rather than the specific training paradigm, is the key factor, we should observe similar effects across different approaches to improving model reasoning capabilities.

To test this hypothesis, we conduct two additional experiments that examine models that enhance reasoning through alternative methods. Our experimental setup involves two comparisons. First, we compare the standard Qwen2.5-7B-Instruct model with DeepSeek-R1-Distill-Qwen-7B that has been distilled from DeepSeek-R1 (Guo et al., 2025), inheriting its advanced reasoning capabilities. Second, we evaluate models from the Qwen3 series (8B and 32B) (Yang et al., 2025), which feature a native "thinking" mode that can be enabled or disabled. For each model and configuration, we measure the hallucination rates on the No-Tool-Available Task (NTA) and Distractor-Tool Task (DT) of our SimpleToolHalluBench.

As shown in Table 1, models with enhanced reasoning consistently exhibit higher hallucination rates. The R1-distilled model shows marked increases, indicating hallucination tendencies transfer via distillation. Similarly, enabling “thinking” mode in Qwen3 models raises hallucination on both tasks. These findings confirm that the link between enhanced reasoning and increased hallucination is method-agnostic.

These findings strongly indicate that the link between enhanced reasoning and increased hallucination is a widespread phenomenon. The very cognitive processes that allow models to perform complex reasoning, whether trained via RL, inherited through distillation, or natively activated, also appear to make them more prone to fabricating unsupported information in fallible, tool-related scenarios. Additional results covering a broader range of model scales are provided in Appendix B.

4.4 Isolating Reasoning as the Driving Factor

The experiments in Sections 4.1–4.3 establish a consistent association between reasoning enhancement and increased tool hallucination. However, two alternative explanations remain: (i) the effect could reflect RL training itself rather than the reasoning it induces, and (ii) it could be a byproduct of general instruction-following degradation known to accompany post-training. In this subsection, we present two controlled experiments that address these alternatives directly.

4.4.1 Ablating the Reasoning from RL Training

To disentangle the contribution of the reasoning from that of RL training itself, we train Qwen2.5-7B-Instruct on SynTool under two regimes that differ only in whether an explicit reasoning step is required: (i) direct tool-use RL, in which the model is trained to invoke tools without producing any block, and (ii) think-then-act RL (standard ReCall setup). Both regimes share the same training data, reward function, optimizer, and hyperparameters.

As shown in Table 2, removing the reasoning step yields only a moderate increase in hallucination (: 34.8 41.4), whereas enforcing think-then-act RL produces a dramatic jump to 90.2. Since the two regimes are matched in every respect except the reasoning step, the sharp gap indicates that the observed amplification is most closely associated with the reasoning step itself, rather than with RL training in general.

4.4.2 Ruling Out General Instruction-Following Degradation

A second alternative explanation is that reasoning RL broadly degrades instruction-following ability, and that tool hallucination is merely a symptom of this more general regression. To test this, we evaluate ReCall-7B on two instruction-following benchmarks: IFEval (Zhou et al., 2023) for basic instruction compliance and ComplexBench (Wen et al., 2024) for multi-constraint complex instructions. We also include the BFCL Multi-Turn (Patil et al., 2025) base subset as a complement to the SynTool validation reward, providing an independent measure of general-purpose tool-calling competence on held-out tools and scenarios.

As shown in Table 3, instruction-following ability remains largely stable after reasoning RL (IFEval: ; ComplexBench: ), and tool-calling competence on BFCL Multi-Turn actually improves substantially (). Yet tool hallucination on SimpleToolHalluBench simultaneously surges (: ; : ). The fact that improved tool-use competence coexists with amplified tool hallucination demonstrates that tool hallucination is a distinct failure mode not captured by existing instruction-following or tool-calling benchmarks, highlighting the unique diagnostic value of SimpleToolHalluBench.

5 Mechanistic Analysis

Our experiments reveal a paradox: Reasoning RL increases tool hallucination even when trained on non-tool tasks (Section 4). This counterintuitive finding demands a mechanistic explanation. We conduct a two-stage analysis: first examining how Reasoning RL reshapes the model’s representation space, then localizing where hallucination emerges within the reshaped model.

5.1 Representation Collapse: Reasoning RL Destabilizes Tool Pathways

The fact that non-agentic Reasoning RL increases tool hallucination suggests that the optimization process causes collateral damage, enhancing targeted reasoning capabilities while inadvertently disrupting other domains. We hypothesize that Reasoning RL, regardless of the specific training task, is associated with disproportionately larger representational shifts on tool-related queries.

To test this hypothesis, we conducted a controlled experiment using the Qwen2.5-7B-Instruct model. We fine-tuned it using Group Relative Policy Optimization (GRPO) on math reasoning task(GSM8K), then compared the internal representations of this post-RL model against the original base model. To quantify the representational changes, we employed Centered Kernel Alignment (CKA), a robust similarity metric for neural representations. CKA measures the similarity between two representation matrices and by computing:

where and are Gram matrices, and HSIC is the Hilbert-Schmidt Independence Criterion. CKA ranges from 0 (completely dissimilar) to 1 (identical representations). We evaluated these changes on two distinct test sets: in-distribution samples from the training domain and OOD tool-calling problems from SimpleToolHalluBench. This design allows us to assess whether Reasoning RL produces asymmetric representational shifts, preserving representations relevant to the training task while inducing substantially larger drift on inputs related with tool hallucination.

Figure 4 reveals a striking asymmetry that confirms our hypothesis. In-distribution representations remain highly stable across all layers (CKA 0.9), indicating that the RL process preserves and refines the pathways relevant to its training objective. However, tool-related representations show dramatic collapse, with CKA scores plummeting below 0.75 in early and middle layers, precisely where initial feature extraction and reasoning patterns form. This asymmetry suggests that Reasoning RL does not merely enhance targeted capabilities but substantially reshapes the model’s representation space. The model’s representations become more specialized toward the training domain, while representations on tool-related inputs drift more substantially from the base model, a pattern that co-occurs with the increased hallucination rates we observe on SimpleToolHalluBench. Further mechanistic analysis of the CKA collapse is detailed in Appendix E.

5.2 Localizing Activation Differences

Having established that Reasoning RL disrupts tool representations globally, we now examine which model components exhibit the strongest activation differences when comparing correct tool usage versus hallucination. This analysis aims to identify the neural correlates of hallucination behavior within the RL-trained model.

We analyzed activation patterns within the GRPO-trained model when processing identical queries that elicit either correct or hallucinated responses. For each architectural component at each layer, including attention output (attn_out), MLP output (mlp_out), and the residual stream at two points (resid_mid and resid_post), we quantified the distinguishability between these two response types.

Formally, for a given component, let be the set of activation vectors from correct responses and be the set from hallucinated responses. We construct a labeled dataset where each is paired with its label . We then train a linear classifier . To quantify the distinctiveness of the representations beyond chance, we define the discrimination score as the classification accuracy’s gain over a random baseline ():

| (2) |

This discrimination score measures the linear separability of activation patterns relative to a random guess. A score of implies that the activation patterns are indistinguishable (no better than a coin flip), while higher positive scores indicate that a component’s activations differ systematically between correct and hallucinated behaviors.

Figure 5 reveals a clear pattern: residual stream components, particularly from layer 20 onward, exhibit discrimination scores exceeding 0.14, significantly higher than attention (avg. 0.02) and MLP (avg. 0.04) outputs. This concentration in late-layer residual streams highlights where the divergence between correct and hallucinated responses becomes most apparent.

This finding aligns with the view of the residual stream as the primary pathway for accumulating information (Elhage et al., 2021). While individual attention and MLP modules process inputs similarly (yielding low discrimination scores), their cumulative contributions produce increasingly divergent trajectories in the residual stream. Small, initially imperceptible differences compound during propagation, manifesting in late layers as distinct activation patterns correlated with the eventual correct or hallucinated output.

The concentration of discriminative signals in the residual stream, rather than within individual computational modules, is consistent with hallucination arising from the gradual accumulation of subtle processing differences rather than a discrete failure in any single component. Consequently, effective mitigation may need to either prevent initial divergence in early layers or directly intervene on the accumulated signal in late-layer residual streams.

6 Is There a Free Lunch in Mitigating Tool Hallucination?

Our analysis reveals both the scope of the reasoning-hallucination problem (Section 4) and insights into its mechanistic origins (Section 5). This naturally raises a practical question: can existing alignment techniques effectively mitigate these failures without severely compromising the agent capabilities we sought to enhance? We investigate this critical question by evaluating two widely-used approaches: prompt engineering, which tests whether explicit instructions can override learned hallucination tendencies, and Direct Preference Optimization (DPO), which examines whether post-hoc preference learning can correct the reliability issues while preserving utility.

6.1 Methodology

Both mitigation experiments are conducted on the ReCall-7B model Chen et al. (2025), as shown in our earlier experiments, exhibits a heightened hallucination rate post-RL training. This serves as a challenging baseline to evaluate the effectiveness of our mitigation strategies.

Prompt Engineering.

This approach guides the model using explicit instructions. For both the No-Tool-Available (NTA) and Distractor-Tool (DT) tasks, where necessary tools are absent, we augment the system prompt with a command: “You must not use any tools that are not explicitly provided to you.” This tests whether a simple instruction can override the model’s ingrained hallucination tendency. The full prompt is in Appendix A.2.3.

Direct Preference Optimization (DPO).

This method fine-tunes the model to prefer “honest” responses over “hallucinated” ones using a preference dataset with two scenarios:

• 1. When the necessary tool is unavailable: The chosen response honestly admits the inability to call the required tool. The rejected response fabricates the non-existent tool and its output.
• 1. When the necessary tool is available: To prevent passivity, the chosen response correctly invokes the tool. The rejected response evasively refuses to answer despite having the capability.

By training on these preferences, we aim to instill a policy that defaults to honesty when tools are absent.

6.2 Results and Analysis

We evaluate each mitigation strategy on our SimpleToolHalluBench to measure changes in hallucination rates ( and ). Crucially, we also re-evaluate the model’s performance on the SynTool validation set to measure any degradation in its core tool-using utility. The results are summarized in Table 4.

The experimental results lead to two key observations. First, prompt-based methods offer minimal mitigation. While adding an explicit instruction slightly reduces the hallucination rate, the effect is marginal. The model largely ignores the directive, suggesting that the behavior learned during RL is too deeply ingrained to be corrected by a simple prompt. This indicates that prompt engineering may not be a sufficient solution.

Second, DPO alignment is effective but incurs a utility cost. The DPO-aligned model shows a substantial reduction in hallucination on both the NTA and DT tasks, confirming that preference tuning can successfully teach the model honesty. However, this improvement comes at a price. The model’s validation reward on SynTool, a measure of its general tool-reasoning capability, sees a significant drop. This trade-off suggests that in learning to be more cautious and honest, the agent becomes less effective at proficiently using tools even in appropriate scenarios. While DPO presents a promising direction, further research is needed to mitigate hallucination without sacrificing essential agentic utility.

7 Conclusion and Outlook

This work uncovers a fundamental paradox in evolving LLM agents: techniques that enhance reasoning capabilities, whether via RL, distillation, or toggleable modes, are consistently accompanied by reduced tool-use reliability and increased hallucination. Controlled ablations show that this effect is most closely associated with reasoning itself rather than RL training in general, and cannot be attributed to broader instruction-following degradation. Our mechanistic analysis reveals that reasoning RL induces disproportionately larger representational shifts on tool-related inputs, with late-layer residual streams emerging as where correct and hallucinated responses become most linearly separable. We further identify a severe reliability-capability trade-off: improved honesty via methods like DPO comes at the direct expense of agent utility. Achieving trustworthy agents thus requires moving beyond isolated reasoning scaling toward objectives that explicitly co-optimize for confidence calibration and abstention, ensuring that “thinking deeper” does not imply “acting less reliably.”

Limitations

This work has several limitations that should be considered when interpreting our results. First, our benchmark SimpleToolHalluBench focuses on single-step tool invocation scenarios; real-world agentic systems often involve multi-step tool chains where hallucination effects may compound or manifest differently. Second, our mechanistic analysis, while revealing important patterns in representation collapse and activation divergence, does not yet provide a complete causal account of why reasoning enhancement specifically destabilizes tool-related pathways. Finally, our mitigation experiments are limited to prompt engineering and DPO; other techniques such as process supervision, constitutional AI approaches, or novel reward shaping methods remain unexplored.

Ethical Considerations

This work studies tool hallucination in reasoning-enhanced LLM agents and highlights a reliability risk introduced by current reasoning optimization techniques. By demonstrating that stronger reasoning can increase the tendency to fabricate or misuse tools, our findings raise concerns for deploying agentic systems in real-world settings where incorrect tool usage may lead to misleading or harmful outcomes.

Our benchmark and experiments rely exclusively on synthetic queries, public tools, and open-source models, and do not involve human subjects or sensitive personal data. While our analysis may inform the design of safer agents, it does not provide direct safeguards for high-stakes applications. Deployments in domains such as finance, healthcare, or infrastructure should incorporate additional protections, including explicit abstention mechanisms, human oversight, and stricter validation of tool calls.

We hope this work encourages the community to jointly consider capability and reliability when developing future reasoning-oriented language agents.

References

• Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470. Cited by: §2, Figure 2, §4.1, §6.1.
• Investigating truthfulness in a pre-release o3 model. Note: https://transluce.org/investigating-o3-truthfulness Cited by: §1.
• Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Figure 3, §4.2.
• A mathematical framework for transformer circuits. Transformer Circuits Thread. External Links: Link Cited by: §5.2.
• Honestllm: toward an honest and helpful large language model. arXiv preprint arXiv:2406.00380. Cited by: §2.
• DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §A.4, §2, §4.3.
• Search-r1: training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Cited by: §2.
• A survey on the honesty of large language models. arXiv preprint arXiv:2409.18786. Cited by: §2.
• Torl: scaling tool-integrated rl. arXiv preprint arXiv:2503.23383. Cited by: §2.
• OpenAI o3 and o4-mini system card. Note: https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf Cited by: §1.
• The berkeley function calling leaderboard (BFCL): from tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §4.4.2.
• Gorilla: large language model connected with massive apis. Advances in Neural Information Processing Systems 37, pp. 126544–126565. Cited by: §1.
• Toolrl: reward is all tool learning needs. arXiv preprint arXiv:2504.13958. Cited by: §2.
• Toolformer: language models can teach themselves to use tools. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.
• SEM: reinforcement learning for search-efficient large language models. arXiv preprint arXiv:2505.07903. Cited by: §1.
• Agent safety alignment via reinforcement learning. arXiv preprint arXiv:2507.08270. Cited by: §2.
• DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2, §4.2.
• R1-searcher: incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592. Cited by: §2.
• Learning to summarize from human feedback. Advances in Neural Information Processing Systems 33, pp. 3008–3021. Cited by: §2.
• Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014–10037. Cited by: §1, §2.
• Otc: optimal tool calls via reinforcement learning. arXiv e-prints, pp. arXiv–2504. Cited by: §2.
• Ragen: understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. Cited by: §2.
• Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, pp. 24824–24837. Cited by: §1, §2.
• Benchmarking complex instruction-following with multiple constraints composition. Advances in Neural Information Processing Systems 37, pp. 137610–137645. Cited by: §4.4.2.
• Reducing tool hallucination via reliability alignment. arXiv preprint arXiv:2412.04141. Cited by: §2.
• Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.3.
• React: synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.
• Siren’s song in the ai ocean: a survey on hallucination in large language models. Computational Linguistics, pp. 1–46. Cited by: §2.
• ToolBeHonest: a multi-level hallucination diagnostic benchmark for tool-augmented large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11388–11422. Cited by: §2.
• Agent-safetybench: evaluating the safety of llm agents. arXiv preprint arXiv:2412.14470. Cited by: §A.1, §3.1.
• Deepresearcher: scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160. Cited by: §2.
• Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: §4.4.2.

Appendix A Details and Examples of the SimpleToolHalluBench

A.1 The Details of the Construction of SimpleToolHalluBench.

We construct the benchmark as follows: We began by selecting 349 diverse tools from Agent Safety Bench (Zhang et al., 2024b), deliberately covering broad scenarios including web search, operating system operations, transportation, financial transactions, healthcare, and scientific computing to ensure systematic coverage. For each tool, we used ChatGPT-4o to generate two distinct types of user queries, mapping to our two experimental settings:

• 1. No-Tool-Available Task (NTA): We generate Explicit Invocation Queries where the user directly names the required tool (e.g., “Please use the get_restaurant_address tool to find…”). In this setting, the system prompt contains no tools, testing whether the model can abstain from hallucinating non-existent tools.
• 1. Distractor-Tool Task (DT): We generate Implicit Requirement Queries where the user describes a task that necessitates a specific tool without naming it (e.g., “Find the current occupancy rate…”). In this setting, only an irrelevant distractor tool is provided, testing whether the model correctly identifies that the available tools are insufficient.

To ensure valid evaluation, we applied strict manual quality control. After generation, all candidate queries were reviewed by two independent annotators (with a third reviewer resolving disagreements). We explicitly filtered out queries where the answer could be partially derived from the model’s internal knowledge (e.g., common facts or generic advice), ensuring our hallucination labels reflect genuine fabrication rather than helpful, knowledge-based responses. This process resulted in a final set of tool-query pairs. We emphasize that these 296 tools provide representative coverage across functional domains, making this lightweight benchmark sufficiently effective for the targeted diagnostic goal of evaluating tool hallucination tendencies.

A.2 System Prompts Used for Evaluation

For the No-Tool-Available Task, we use a minimal system prompt that provides no tools while the user’s query requires an external tool, simulating a setting where the agent truly lacks tool access. For the Distractor-Tool Task, the agent is equipped with tools, but none are appropriate for the user’s request; the available tools are plausible yet irrelevant.

A.2.1 System Prompt for the reasoning model

A.2.2 System Prompt for the non-reasoning model

A.2.3 System Prompt Example for the mitigation experiments

A.3 Query Examples and Labeled Responses

A.4 Details of LLM-as-Judge Evaluation

We employ DeepSeek-R1 (Guo et al., 2025) as the judge to determine whether a model response exhibits tool hallucination. The full system prompt used for this evaluation is provided below. To ensure reliability, we manually inspected a subset of the judgments and found that the vast majority were consistent with human evaluation, confirming the robustness of the LLM-as-Judge protocol.

Appendix B Additional Experimental Results on SimpleToolHalluBench

To further validate the universality of the Reasoning Trap, we extended our evaluation on SimpleToolHalluBench to a broader set of state-of-the-art model families, including Qwen3, DeepSeek, and Kimi. Specifically, for each family, we compare the standard Instruction-Tuned model against its corresponding Reasoning-Enhanced variant (denoted as “Thinking” or “R1”). These reasoning variants are typically trained from the instruction-tuned baselines to specifically enhance chain-of-thought capabilities.

Table 5 summarizes the hallucination rates ( and ). The results consistently reinforce our core finding: the transition from standard instruction tuning to reasoning-intensive training invariably amplifies tool hallucination.

Consistent Degradation Across Scales and Families.

The experimental results reveal a uniform trend: enhancing reasoning capabilities comes at the cost of tool reliability. For the Qwen3 family, the “Thinking” versions show significantly higher hallucination rates on the No-Tool-Available () task compared to their Instruct baselines, with the 4B model showing a nearly 9-fold increase (3.4% to 29.4%).

Crucially, this phenomenon persists even at the largest scales. DeepSeek-R1 (671B), a specialized reasoning model, exhibits higher hallucination rates ( ) compared to its base instruction-tuned counterpart, DeepSeek-V3 (). Similarly, the Kimi-K2-Thinking model shows increased hallucination compared to the standard Kimi-K2. These findings indicate that the trade-off between reasoning depth and tool fidelity is a fundamental challenge that scaling parameters alone does not resolve.

Appendix C Algorithmic Details for GRPO and DPO

C.1 Group Relative Policy Optimization (GRPO)

GRPO removes the need for a learned value function by computing relative advantages within groups of sampled responses for the same prompt.

Setup.

For a prompt , sample rollouts and obtain scalar rewards . Define the group mean and (optionally) the group std .

The group-relative advantage is

Training Objective.

Let be the per-token probability ratio, defined as:

With PPO-style clipping and a token-level KL to a frozen reference policy , the GRPO loss is:

| (3) |

where is the token context, is the clip range, and controls conservatism. Sequence-level variants average token terms or add explicit length normalization.

C.2 Direct Preference Optimization (DPO)

DPO directly optimizes a pairwise preference model without explicit reward modeling or RL rollouts.

Setup.

Training data consists of where is preferred over . Let be a frozen reference policy. Define sequence log-likelihoods and (optionally length-normalized). DPO maximizes the probability that the model prefers over after subtracting the reference logits.

Loss.

The standard DPO objective is

| (4) |

where and controls sharpness. Intuitively, DPO pushes the log-odds of vs. beyond the reference margin.

Gradient Intuition.

Let . Then

| (5) |

which increases the relative score of vs. until the model’s margin exceeds the reference-adjusted boundary.

Our Preference Construction.

chosen = honest abstention when required tools are missing; or correct tool usage when available. rejected = fabricated tool calls/outputs; or needless refusal when capable. This teaches abstention and preserves competence when tools are present.

Appendix D Details of ReCall: Learning to Reason with Tool Calls via RL

Overview. ReCall is a simple framework for teaching LLMs to interleave natural-language reasoning with tool calls using reinforcement learning (RL). Instead of relying on step-level or process supervision, ReCall treats the model as an agent that interacts with an external tool environment in a closed loop: the policy generates a reasoning trace that may include one or more structured tool invocations, the environment executes those tools and returns results, and the model continues reasoning conditioned on these results. The optimization objective is purely outcome-based (task success/quality), so the model is free to discover its own patterns of when and how to call tools as part of its reasoning strategy.

Training data and tasks.

The original ReCall framework trains on a mixture of (i) a synthetic multi-tool dataset SynTool and (ii) real multi-hop QA tasks that are explicitly designed to require external tools (e.g., search or retrieval). Each SynTool instance specifies a set of available tools with their signatures, a natural-language query whose solution requires composing those tools, and a deterministic environment that produces the ground-truth answer when the correct sequence of tool calls is executed. This design allows ReCall to generate diverse multi-step tool-use trajectories while keeping the underlying environment fully controllable. In our reproduction, we train only on SynTool. This isolates the effect of tool-reasoning RL itself, without confounds from additional real-world QA datasets, and ensures that any observed changes in hallucination behavior are driven by the GRPO training on synthetic tool-use tasks rather than distributional shifts in the data mixture.

Optimization recipe and stack.

ReCall uses a standard GRPO-style RL loop (see Appendix C.1) on top of an instruction-tuned base model such as Qwen2.5-7B-Instruct, served via a high-throughput inference stack (e.g., vLLM/SGLang). For each training step, the GRPO pipeline samples prompts from SynTool, rolls out the current policy in the tool-execution environment until termination (producing a reasoning trace with any intermediate tool calls and tool outputs), and assigns a scalar reward based on task outcome (e.g., correctness and answer quality). Within each prompt group, GRPO computes group-relative advantages and applies a PPO-style clipped policy gradient with a KL penalty to a frozen reference model.

In our implementation, the policy emits structured tool-call blocks (name + arguments) inside the generated text; these are parsed and executed by a sandboxed tool runner, and the returned tool outputs are appended to the context as additional observations before the next decoding step. Training periodically saves checkpoints and evaluates them under the same closed-loop tool environment on the SynTool validation split. These checkpoints are exactly the ones we later evaluate on SimpleToolHalluBench in Section 4.1, allowing us to track how GRPO-based tool-reasoning optimization simultaneously affects task reward and tool hallucination rates.

Appendix E Additional CKA Analyses

In this appendix, we provide additional analyses of how Reasoning RL (GRPO on SynTool via ReCall) affects model representations, complementing the layer-wise CKA results in Section 5.1.

E.1 Module-Level CKA: Attention vs. MLP vs. Residual Stream

The layer-wise CKA curves in the main text are computed on the residual stream. For each transformer block , we take the post-block residual activations, after adding both the attention and MLP sublayer outputs back to the residual, and measure CKA similarity between the pre-RL and post-RL models. The “collapse” emphasized in the main paper is therefore a property of the integrated residual representation that aggregates contributions from all attention heads and the MLP within each block.

To probe modules more directly, we additionally compute CKA on the attention output and the MLP output of each layer before they are added back into the residual stream. Concretely, for each block we extract the output of the multi-head attention sublayer and the output of the MLP/swiGLU sublayer, and compute CKA between the pre-RL and post-RL models for these tensors separately. The resulting layer-wise curves are shown in Figure 6.

The attention-output and MLP-output CKA trends closely follow the residual-stream curve. In all three cases, CKA decreases smoothly with depth rather than exhibiting sharp drops localized to a few isolated blocks. A natural explanation is that both sublayers take the previous residual stream as input; once Reasoning RL induces a global drift in residual representations, this drift propagates through both modules in a distributed fashion. We also observe that MLP CKA is consistently lower than attention CKA, suggesting that MLPs are somewhat more strongly updated by RL, but this effect remains spread across layers instead of being concentrated in a small set of attention heads or blocks.

E.2 Cross-Domain CKA: SynTool vs. GSM8K

We further compute CKA between the base model and the SynTool+ReCall (GRPO) model on two distinct input domains, using the same protocol as in Section 5.1. The first domain consists of SynTool-style tool inputs drawn from the synthetic tool-reasoning benchmark used for RL training. The second domain consists of GSM8K math reasoning inputs, representing a standard reasoning domain without tools. For each domain, we compute layer-wise CKA on the residual stream between the base and post-RL models. The resulting curves are shown in Figure 7.

Both domains exhibit a pattern similar to that in Figure 6. For SynTool queries and for GSM8K queries, CKA drops sharply in the first few layers, down to approximately –, with only a modest gap between domains. We do not observe an inverted pattern in which tool representations remain almost unchanged while reasoning representations collapse. Instead, the two domains show comparable early-layer drift, and their CKA trajectories largely track each other over depth.

This analysis shows that the GRPO+SynTool agent RL setting does not carve out a “safe” in-distribution regime for tool queries. Despite being the direct training domain, SynTool inputs still undergo substantial representational drift relative to the base model. RL focused solely on tool utility therefore does not resolve reliability concerns; it can push tool-related representations away from the base along exactly the dimensions where tool-reliability queries, such as those in SimpleToolHalluBench, become more fragile.