Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
Computer-use agents are increasingly capable of operating desktops, browsers, and productivity software through screenshots, keyboard input, and mouse actions. Yet a long task can fail because of one early mistake: a misplaced click, an incorrect selection, or an unnoticed change to application state. Once that error propagates, simply continuing may create more side effects. Restarting from the beginning is safer, but often wasteful and impractical in environments that cannot be reset.
In Lost in Recovery, Byeonghun Bang, Sangwoo Park, Jinheon Baek, Kangsan Kim, Youngwan Lee, and Sung Ju Hwang study whether computer-use agents can recover from these failures. They introduce RETRACE, an evaluation framework for measuring autonomous, model-based recovery rather than treating rollback as an informal engineering trick.
Why recovery is different from reflection or restart
The authors distinguish recovery from several familiar responses to failure. Reflection allows the agent to recognize a mistake and continue from the current state, which can leave unintended changes in place. A full rollback returns to the initial environment and replays the entire task, but this can be expensive or unavailable in proprietary operating systems and websites. API-based undo mechanisms are useful in some web settings, but do not generalize cleanly to desktop applications.
Recovery instead aims to restore a prior intact state and resume from there. The challenge is that a screenshot may not reveal everything that matters. Two screens can look identical while differing in a hidden configuration, selection, focus state, or file-side effect. The paper therefore argues that visual similarity alone is not a sufficient definition of recovery.
RETRACE and transition-consistent recovery
RETRACE defines successful restoration through what the authors call Transition-Consistent Recovery. A recovered state matches its target when the same downstream actions produce the same outcomes. This behavioral criterion is intended to capture hidden state as well as visible appearance.
The framework builds its evaluation data in three stages:
- On-policy rollout: the researchers collect successful trajectories generated by each evaluated agent. They replay action sequences repeatedly and retain trajectories whose outcomes remain consistent.
- Trajectory drift: an intermediate state is selected as the recovery target. A perturbation policy then intentionally moves the environment off the task path, creating a controlled error state.
- Goal-oriented action replay: a recovery policy acts from the error state. Instead of requiring exhaustive state annotations at every step, RETRACE replays the remaining actions from the original successful trajectory and checks whether the original task outcome is reproduced.
The final evaluation combines transition consistency with observation-level matching. This lets the framework reject both visually mismatched states and states that look right but behave differently when the task continues.
What the experiments show
The authors evaluate RETRACE on OSWorld and WindowsAgentArena using closed- and open-source computer-use agents. Their preliminary ideal-recovery experiment provides the clearest evidence that recovery matters: when a failed trajectory is resumed from an appropriate earlier state, it performs better than restarting from the initial state under the same retry budget. The advantage is especially visible when only a small number of attempts is available.
The main recovery experiments expose a gap between latent ability and routine behavior. Under implicit recovery, the forward agent simply continues with no recovery-specific instruction. The evaluated systems generally recover poorly, including strong models. The paper reports that EvoCUA-32B has an average recovery rate below 1% in the highlighted results.
Explicitly instructing a model to restore a target state produces a substantially different picture. Across the Linux and Windows environments, explicit recovery outperforms implicit recovery by an average margin of 17.86 percentage points, according to the authors. The largest improvements appear among more generalist agents, while some post-trained specialist models remain weak. The authors interpret this as evidence that recovery capability is often latent but not reliably elicited during ordinary task execution.
The evaluation protocol itself also receives a human check. In a study involving 12 annotators and 150 evaluation samples, RETRACE agreed with human judgments more often than the tested alternatives, including image-only and transition-only variants and a separate model evaluator. The reported average inter-annotator agreement was 82%, with a Cohen’s kappa of 0.45.
Recovery can help—or make failure worse
End-to-end results show why recovery quality matters operationally. For Qwen3.5-9B, model-based recovery improved task success over the no-recovery baseline across the tested recovery-context settings, with an average gain of 2.12 percentage points. However, recovery was not universally beneficial. UI-TARS-1.5-7B and EvoCUA-32B showed lower task success in one contextualized-recovery comparison, illustrating that an inaccurate restoration can introduce additional drift.
The qualitative analysis identifies two recurring failure modes. In spurious perception, the agent fixates on a visible element—such as a hovered button—that is present in the incorrectly restored state and acts on it as though it were relevant. In state-tracking failure, the agent assumes that the environment has returned to the target state even when an application panel, selection, or other operational detail is different.
Limitations and practical implications
RETRACE is an evaluation framework, not a complete recovery solution. The authors emphasize that explicit prompting can only elicit capabilities already present in the underlying model. It cannot compensate for fundamental weaknesses in GUI perception, state tracking, or action planning. The benchmark construction also relies on controlled environments, filtered trajectories, oracle task evaluators, and replayable action sequences, so real production systems may introduce additional nondeterminism.
For developers, the practical lesson is to treat recovery as a first-class capability rather than an afterthought. Agents should be evaluated on whether they can restore a functionally equivalent state, not merely produce a similar screenshot. Recovery tests should include hidden side effects, interrupted workflows, and limited retry budgets. At the same time, automated recovery should be guarded: a weak recovery policy may be more dangerous than a transparent restart.
The authors suggest recovery-aware post-training on erroneous states and corrective trajectories, alongside objectives for tracking latent state and predicting action-induced transitions. Those methods remain future work. What RETRACE demonstrates is narrower but important: reliable computer-use agents need not only to choose the next action, but also to know when—and how—to safely return to a state from which the task can still succeed.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.