Why deep-search agents need a stress test
Search agents are increasingly designed to retrieve information over several steps, revise queries, and synthesize an answer from external sources. Yet a high final-answer accuracy score can conceal an important weakness: an agent may confidently trust evidence that is irrelevant, factually wrong, or visibly unreliable.
In DeepStress: Stress-Testing Deep Search Agents, Ismaël Rousseau, Géraldine Damnati, and Frédéric Béchet propose a controlled way to expose that weakness. Their framework replaces an agent’s usual search tool with a simulated environment that dynamically generates documents for each query. The researchers can then vary three properties independently: trustworthiness, or whether a source appears credible; relevance, or whether it addresses the query; and factuality, or whether its claims are correct.
The goal is not to replace ordinary benchmarks. Instead, the authors present DeepStress as a laboratory for asking how an agent behaves when the evidence pipeline starts to fail.
How DeepStress works
For every search call, the simulator samples document attributes according to the selected scenario. A document may be relevant and accurate but come from an untrustworthy-looking domain, or it may look credible while containing severe factual errors. Relevant factual documents are grounded in supporting material from HotpotQA or BrowseComp-Plus. Irrelevant documents are generated from another question in the dataset, while incorrect documents are prompted to contain factual errors.
This dynamic design matters because different agents can take different search paths. Rather than returning a fixed set of pre-generated results, DeepStress creates a result for the query the agent actually issued. The paper reports a human sanity check of 50 documents: intended labels matched human judgments particularly well for trustworthiness, while factuality was more difficult to assess consistently.
The evaluation covered 12 systems, two datasets, seven quality scenarios, and 200 randomly sampled questions per dataset. The authors varied one reliability dimension at a time, testing quality levels of 0%, 50%, and 100%, with a maximum of 100 tool calls per question.
Accuracy is not the whole story
DeepStress classifies runs into five outcomes: correct, incorrect, abstention, exceeding the tool budget, or failing to produce the expected answer format. This is more informative than treating every unsuccessful run as the same kind of failure.
The authors also introduce the Reliability-Aware Score, or RAS. It gives full credit to correct answers, no credit to incorrect or malformed answers, and partial credit to abstention-like behavior when the available evidence is unlikely to be reliable. Cost per Reliability-Aware Score, or CoPRAS, adds computational cost to that picture. The paper emphasizes that these are design choices, not universal measures, but argues that they better distinguish cautious failure from overconfident failure.
What the experiments found
The results show substantial differences between systems. Several open-weight agents, including Search-R1, changed little when trustworthiness was degraded, continuing to rely on documents even when the sources carried obvious credibility red flags. Search-R1’s aggregated correct-answer rate stayed around 78% to 80% across trustworthiness levels, with no clear corresponding increase in abstention.
Other systems responded to low-quality evidence by searching longer or declining to answer. For example, the paper reports that GPT-5-mini at high reasoning effort increased tool-budget exhaustion from 32.0% under fully trustworthy evidence to 62.7% when trustworthiness was reduced to 0%. DR-Tulu’s correct-answer rate fell from 59.8% to 40.9% as trustworthiness declined, while missing-answer outcomes increased.
Relevance degradation was particularly damaging. Across systems, correct answers fell sharply when returned documents did not address the question. Search-R1’s incorrect-answer rate rose from 19.5% at full relevance to 79.2% at zero relevance in the reported aggregate results. AutoRefine showed a different brittleness: missing answer tags increased substantially as relevance declined.
Factually wrong but relevant documents were also dangerous. The authors argue that irrelevant documents can be easier to reject than relevant documents containing false claims. At zero factuality, DR-Tulu reached 59.1% incorrect answers in the aggregate analysis, compared with 33.8% at zero relevance.
Cost rose as evidence quality worsened. Average run cost increased from 10.9k to 20.5k in the paper’s cost units, while average correctness fell from 0.66 to 0.30. RAS fell less sharply, to 0.61, because it credited some cautious behavior. This illustrates the distinction between a system that becomes wrong and one that recognizes that the evidence is inadequate.
The benchmark contamination warning
The contrast between the datasets is one of the paper’s most important findings. When factuality was set to 0%, agents still answered many HotpotQA questions correctly, despite receiving no accurate supporting evidence. GPT-5-mini at medium reasoning effort reached 60% correctness in that setting. On BrowseComp-Plus, performance fell almost to zero.
The authors interpret this as evidence that many HotpotQA questions can be solved from parametric knowledge, likely helped by the familiarity of Wikipedia-derived content. Their direct-inference baseline supports that interpretation: systems answered a substantial fraction of HotpotQA questions without search, while BrowseComp-Plus performance was generally close to zero.
This does not prove that every correct answer under corrupted evidence is a failure. It may represent useful internal knowledge. But the result shows why search-agent evaluations need a no-search baseline and should test conflicts between retrieved evidence and model memory.
Limitations and practical implications
DeepStress simplifies real search. Each call returns one synthetic document, rather than a ranked collection containing mixtures of incomplete, contradictory, and irrelevant results. The experiments use one search loop and one tool, so the findings may not transfer directly to systems with rerankers, summarizers, sub-agents, or more complex orchestration.
The documents are also LLM-generated and the three reliability dimensions are varied independently. Real web failures often combine low credibility, poor relevance, and factual error. The human validation sample was limited, and factuality judgments were less consistent than trustworthiness judgments. Finally, RAS rewards abstention and tool-budget exhaustion in related ways even though users may reasonably prefer an explicit, early uncertainty statement.
For practitioners, the message is straightforward: evaluate search agents against controlled evidence failures, not only clean retrieval. Track whether systems notice source problems, reformulate queries, seek more authoritative domains, abstain appropriately, and avoid wasting computation. Report detailed outcome categories alongside accuracy, and include direct-inference baselines when the benchmark may overlap with pretraining data.
DeepStress does not establish that one model family is universally safer or more capable. It does demonstrate that agents with similar clean-benchmark performance can fail in very different ways once evidence quality is manipulated. Reliability-aware stress testing can therefore reveal operational risks that ordinary answer accuracy leaves hidden.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.