Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

Computer-use agents are often trained as if every task should be completed through screenshots, mouse movements, and keyboard input. That approach is general, but it can be inefficient. A terminal command or application library may be a better way to edit a structured file, while the graphical interface remains essential for inspecting live application state.

The Microsoft Mage Team presents Mage-VL-Cowork as a 4B-parameter model designed to combine both channels within one policy. The agent can inspect a desktop, issue GUI actions, run code, and execute terminal commands in the same episode. The central claim is that hybrid competence depends not only on model weights, but also on the tools, prompts, execution scaffold, and training environments used to produce demonstrations.

The problem: teaching an agent when to use pixels or code

Providing an agent with a terminal does not automatically make it use that terminal well. The authors argue that a naive teacher may still click through repetitive edits, modify a file whose contents are cached by a running application, or choose a tool that is technically available but poorly suited to the task. Directly distilling those trajectories risks transferring the teacher’s inefficient habits to a smaller student.

Mage-VL-Cowork treats channel selection as a learned execution policy. GUI interaction is needed for spatial and rendering-dependent state; code and terminal operations are useful for structured files, repository navigation, and verifiable changes. The model, rather than a separate orchestrator, decides when to switch between them.

Evolving the harness before collecting data

The paper’s first major contribution is harness evolution. The authors keep the underlying teacher model fixed while revising its surrounding system: the execution scaffold, tool interfaces, prompts, and terminal environment. Candidate configurations are evaluated for task success, execution cost, feedback from applications, and channel usage.

This process can add useful capabilities, but it can also remove redundant or confusing tools. The paper gives an example in which eliminating an alternate mouse-and-keyboard route avoids competing interaction mechanisms. It also describes adding domain libraries so that a teacher can work directly with slides or other structured artifacts instead of editing lower-level file representations.

In one comparison, a GUI-and-coding harness produced better teacher results than a version that also exposed accessibility tools. On the authors’ held-out comparison, the simpler variant achieved 92.0% teacher success versus 89.6%, with fewer steps. The corresponding 4B student reached 37.6% versus 25.1%. This supports the authors’ conclusion that compact models are especially sensitive to ambiguous tool choices, although the comparison concerns specific harness variants rather than a universal rule about accessibility interfaces.

Reusable workflows replace one-off task construction

To create training environments at scale, Mage-CUA-Forge uses a workflow-first approach. An authoring agent builds and debugs a reusable generation workflow for each atomic skill. Application-specific preparers create files and initial application state, while other workflow components generate instructions, setup scripts, and evaluation rubrics.

Validation gates check both individual artifacts and the consistency of the complete task package. The system then combines skills into composite tasks and rolls them out under GUI-only, terminal-only, and hybrid harnesses. Student evaluation feeds back into the skill taxonomy, allowing the pipeline to add or refine areas where the model performs poorly.

The resulting inventory contains 293,806 synthesized tasks covering 6,915 skills, with 318,837 retained teacher trajectories after execution and quality filtering. The authors report that recurring generation cost fell from $0.906 to $0.158 per task, while token consumption fell from 1,276K to 85K. These figures exclude one-time workflow authoring and debugging, so they describe marginal generation cost rather than the full cost of building the system.

A five-stage training curriculum

Mage-VL-Cowork is trained progressively rather than being exposed to hybrid trajectories immediately:

  1. GUI-oriented midtraining builds visual grounding, structured tool calling, and reasoning formats.
  2. GUI agent SFT teaches multi-step visual interaction.
  3. Coding and terminal SFT develops code generation, debugging, and desktop file manipulation.
  4. Hybrid CUA SFT teaches one policy to route between GUI and terminal actions.
  5. Online hybrid RL refines behavior using verifiable rewards on 1,988 tasks.

The training design also uses a sliding context window: three recent screenshots remain visible while older steps are compressed into concise action summaries. This is intended to preserve evidence of interface changes without retaining every image in a long trajectory.

Reported results

On OSWorld-Verified, the final 4B model scores 66.0% with a 50-step budget, averaged over two runs on 361 held-out tasks. The authors report higher performance than several larger open-source systems listed in their comparison, including 30B and 27B models. The score improves from 45.56% after GUI SFT to 55.84% after hybrid SFT and then to 66.0% after RL.

The model also becomes more efficient across those stages. Mean model-call traces fall from 25.36 to 18.77 per task, while mean output tokens fall from 6,401 after GUI SFT to 5,499 after RL. On the paper’s deterministic WebVoyager-Static benchmark, the hybrid SFT checkpoint reaches 58.6% across 643 tasks.

The authors argue that offline browser evaluation is important because screenshot-only judging can reward plausible answers without proving that the agent visited the required pages, applied all filters, or committed a state-changing action. Their verifier checks execution traces and backend state as well as the final outcome.

Limitations and practical implications

These results are promising, but they should be interpreted within the paper’s evaluation setup. OSWorld comparisons combine systems with different training recipes, budgets, and reporting conventions. The web results use self-hosted replicas, which improve reproducibility but cannot capture every difficulty of live websites. Much of the supervision is synthesized and filtered with automated or model-based judges, creating possible gaps between benchmark success and robust real-world behavior.

The system also requires substantial infrastructure for desktop rollouts, sandboxing, asynchronous execution, and model training. A 4B policy may be attractive for on-device or private deployment, but reproducing the complete data-generation and RL pipeline is not lightweight. Terminal execution additionally requires explicit authorization and isolation.

For practitioners, the clearest lesson is methodological: improve the agent’s action environment before scaling demonstrations. Prune confusing tools, make task outcomes verifiable, reuse environment-building workflows, and train GUI and coding abilities separately before teaching the model to combine them. The paper demonstrates that this recipe can produce a compact hybrid agent; whether the same gains transfer to new applications and unscripted workflows remains an open question.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close