Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Long-horizon agents often fail for a deceptively simple reason: they complete the right subgoals in the wrong order. An agent may place a mug on a table before heating it, or purchase an item before selecting the required options. The failure is not necessarily a lack of knowledge. It can be a failure to anticipate how an action will affect the rest of the task.

The paper by Haochen Liu and colleagues examines whether decision models should be judged only by how well they act. The authors study the same models in three roles: Actor, choosing the next action; Judge, evaluating an action in a recorded trajectory; and Teacher, providing step-level scores during reinforcement learning. Their central finding is that a model can be a poor Actor yet a useful Teacher.

Why acting and judging differ

A decision model produces a choice or a score rather than a long textual response. As an Actor, it sees the task, the interaction history, the current observation, and the available actions. It must choose before the consequences of that choice are visible. This makes prerequisite relationships difficult: an easier, immediately achievable subgoal can attract probability away from the step that must happen first.

The authors tested three dedicated decision models and three general language models used through prompting or constrained readout. On ALFWorld, all six struggled more on tasks with ordered dependencies than on tasks whose subgoals could be completed in either order. JEV, for example, completed 70.8% of tasks with two independent subgoals but only 14.7% of tasks with ordered ones in the authors’ diagnostic comparison.

Removing the easier subgoal from the task changed the models’ behavior substantially. For JEV, the probability assigned to the required state-changing action rose from 14.6% to 77.6%. A general reminder to consider dependencies changed success by only -2.8 to 5.2 percentage points across the models. An ordering example helped some systems, but the effect varied and did not eliminate the problem.

Hindsight turns a weak actor into a better judge

The Judge setting changes the information available. Instead of assessing an action before its effects occur, the model evaluates an action in a recorded trajectory. The authors compare a Prefix context, which ends at the selected action, with a Hindsight context containing the later trajectory. They also compare two questions: whether the task will eventually succeed after the action, called Value, and whether the action made a useful contribution toward the goal, called Progress.

Both choices matter. Seeing the full trajectory helped all six models recognize contributions that were difficult to identify from the prefix alone. For JEV, the mean score for a diagnostic recovery action rose from 0.41 with Prefix to 0.59 with Hindsight.

The question was equally important. When asked about eventual success, JEV gave milestone, other, and no-op actions similar average scores: 0.632, 0.604, and 0.577. When asked whether each action helped, the scores separated to 0.659, 0.440, and 0.298. Across the models, Progress produced larger gaps between useful milestone actions and no-ops. In the paper’s analysis, Value largely repeated the trajectory-level outcome, while Progress supplied more informative credit for individual steps.

Teacher scores improve policy training

The authors use JEV as a fixed Teacher during GRPO training. After each policy rollout, JEV scores the actions using a selected question and context. The scores are centered within the rollout group and added to the outcome advantage, allowing actions in the same episode to receive different learning signals. JEV is not used when the trained policy is evaluated.

The strongest configuration was Hindsight plus Progress. For a Qwen2.5-1.5B policy, success on ALFWorld rose from 72.8% with outcome rewards alone to 82.9%. On WebShop, success increased from 56.8% to 71.4%. The improvement also appeared for the 7B policy, where ALFWorld success rose from 77.6% to 83.8% and WebShop success from 66.1% to 73.6%.

The gains were particularly large on ALFWorld tasks with ordered subgoals. The 1.5B policy trained with Hindsight plus Progress reached 86.8% on those tasks, outperforming the other judging configurations by 6.9 to 16.2 percentage points. It also completed more tasks within shorter action cutoffs in the reported experiments.

Most strikingly, JEV completed 42.9% of the evaluation tasks when acting directly, while the 1.5B policy trained with its scores reached 82.9%. This does not show that a weak judge is universally reliable, but it does demonstrate that direct task success and supervision quality can diverge.

Practical implications for agent builders

  • Evaluate supervisors as judges. A model’s acting score should not be the only criterion for selecting a training supervisor. Test whether it can rank useful and useless actions in recorded trajectories.
  • Ask about contribution, not just outcome. A question about eventual success may assign nearly the same credit to every action. Step-level training needs a question that distinguishes what each action contributed.
  • Use hindsight when possible. The paper supports scoring actions after a rollout, when their consequences are visible, rather than relying only on online prefix information.
  • Screen cheaply before training. Comparing judges and prompts on recorded trajectories can identify promising supervision settings before committing to a full reinforcement-learning run.

Limitations and open questions

The evidence comes from two text-based environments, ALFWorld and WebShop, with episodes capped at 30 and 15 actions respectively. The authors caution that software engineering, tool use, and assistant workflows may involve longer and more varied dependencies. The study also focuses on offline, after-the-episode judging; online filtering or reranking still has only prefix information. Finally, JEV and GPT Decisions are closed systems, so the authors cannot determine which training choices make a decision model an effective judge.

The paper’s demonstrated result is therefore specific but important: on these benchmarks, carefully designed hindsight action judgments improved policy learning even when the judging model itself acted poorly. Whether the same separation between Actor, Judge, and Teacher holds in broader agent systems remains an empirical question.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close