GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning

GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning

Game-playing agents are often tested on simplified environments, offline videos, or tasks whose success is judged by a person. Those settings can miss a central question: can an agent translate visual observations and natural-language goals into reliable actions over a long, changing interaction?

In GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning, Jiajun Chen and colleagues introduce a framework designed to make that question more measurable. GTAGym reconstructs tasks inspired by Grand Theft Auto V, checks outcomes from engine state rather than an agent’s self-reported completion, and generates additional environments for reinforcement learning.

A benchmark built around verifiable gameplay

The evaluation component, GTAVBench, contains 103 online 3D tasks derived from main-story missions and 47 image-based shooting and sniping tasks. The online tasks cover nine categories, including road and on-foot navigation, aquatic and aerial movement, indoor and vertical navigation, following, interception, and evasion. Together with shooting and sniping, the suite spans 11 gameplay dimensions.

The authors manually decomposed 75 missions into subtasks, specifying goals, interaction order, completion constraints, and dynamic behaviors. They then reconstructed scenarios in a web-accessible environment with a screenshot-and-keyboard/mouse interface. A task is successful only when its programmed predicates are satisfied. For example, a following task requires maintaining proximity until the target reaches its destination, while an interception task requires more than merely touching a vehicle: the agent must keep the target blocked under defined motion conditions.

This state-based approach is important because it removes a common ambiguity in game-agent evaluation. Reaching a visually plausible location, or declaring success in text, does not count if the underlying task state is incomplete. The image-based track similarly checks whether shots land inside annotated target regions.

Two ways to test agent capability

GTAVBench supplies a textual instruction alongside screenshots and action history. Its GTAVBench-AGI branch makes the setting more demanding: the objective appears only in the first screenshot, as it might in a game interface, while a harness compresses the growing interaction history.

The authors evaluated six language models under these protocols. The best reported configuration reached 70.9% overall accuracy on GTAVBench-AGI, compared with a mean human accuracy of 89.8%. The result is substantial, but it also leaves a clear gap between current agents and experienced players.

Performance was uneven rather than uniformly weak. The strongest systems handled on-foot, indoor, and vertical navigation relatively well. Dynamic-control tasks were much harder. Aerial navigation, interception, and evasion remained particularly difficult, because success depends on continuously regulating speed, position, heading, and relationships with moving targets.

The paper’s error analysis helps explain why. Across 900 trajectories, the authors identified and categorized 565 first task-relevant errors. Calibration and regulation together accounted for 63.2% of those categorized errors. In practical terms, agents often understood the objective but selected an input that was too strong, too weak, too early, or too late. Examples include overshooting a turn, failing to brake after colliding with a target, descending too rapidly during a landing, or continuing to drive against a wall.

Recovery also separated the models. Within ten subsequent decisions, the reported recovery rate was 65.7% for the strongest configuration and between 21.6% and 25.6% for several other models. This suggests that robust agents need more than perception and planning: they must notice when an action failed and change strategy quickly.

From evaluation to reinforcement learning

GTAGym also includes a template-based generator for training environments. A template fixes the scene, assets, interaction logic, evaluator, and episode budget, while users set ranges or option sets for parameters such as route length, turn count, starting position, movement mode, actor speed, and initial distance.

The generator rejects infeasible configurations and records the resolved settings and seed. Generated episodes expose automatic success, failure, and truncation signals, allowing them to serve as reinforcement-learning environments rather than requiring manual reward labeling. The same design supports batches of 3D tasks and image-based shooting tasks, although the paper’s reported sniping environments are evaluation-only.

In one experiment, the authors trained a Qwen3.8-27B model with LoRA reinforcement learning on 50 generated on-foot navigation instances. On held-out on-foot tasks, accuracy rose from 54.2% to 72.9%. Overall accuracy increased from 34.2% to 38.9%, and the authors also report gains in some unseen categories, including vertical and aquatic navigation. On 100 native GTA V tasks, success increased from 15% to 23%.

These findings support the value of generated training tasks, but they should not be read as proof of broad game competence. Road driving remained at 3.3%, aerial navigation and interception remained at zero, and shooting performance declined in the reported comparison. The transfer result is encouraging, yet still modest.

Limitations and practical implications

GTAGym is a reconstruction of mission-derived gameplay, not the full GTA V world or its complete interaction space. The authors manually designed task decompositions, paths, controllers, and evaluators, so coverage and fidelity depend on those choices. Human evaluation involved three experienced players, and native-game verification was manual, unlike the automated checks used in the reconstructed benchmark.

The benchmark also exposes a systems limitation that matters beyond games: latency. When simulation continued during model inference, accuracy for one tested configuration fell from 62.1% in a paused setting to 39.8%. Following, interception, and evasion fell to zero in that comparison. Shorter action durations improved accuracy in one ablation, but substantially increased interaction rounds and token use.

For researchers, GTAGym offers a useful combination of verifiable online evaluation, long-horizon visual control, parallel execution, and configurable RL generation. For practitioners building computer-use agents, its results point to a concrete priority: improve closed-loop control and recovery, not just instruction following. Agents that can recognize an incorrect maneuver, shorten their action horizon, and adapt to changing motion may be more valuable than agents that simply produce longer plans.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close