AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Many AI-agent evaluations end when a task is completed: a product is found, a form is submitted, or a customer question is answered. The authors of AutoVend argue that seller-side automation needs a harder test. A business agent must make decisions repeatedly, deal with delayed consequences, and turn online actions into profitable physical operations.
AutoVend is a simulated vending operation in which one agent manages 20 machines over 180 days. The machines span factory, campus, residential, commercial, and office locations, all supplied from a shared warehouse. The agent selects products, sets machine-specific prices, places orders, and schedules paid visits to move stock from the warehouse onto shelves.
This setup makes the benchmark more than a product-selection exercise. Procurement takes time, inventory expires, warehouse stock incurs rent, and each machine visit costs money. A decision that looks sensible locally can therefore damage the wider operation by consuming shared inventory, creating unnecessary travel, or tying up cash in slow-moving products.
What AutoVend measures
The benchmark models the task as a partially observable decision problem. The agent sees operating summaries and can query detailed records, but it does not receive the underlying demand parameters or unmet demand directly. It must infer which products suit each location from sales feedback while managing uncertainty.
The simulator contains 1,372 products and is calibrated using six months of order-level records from 99 real machines, covering 220,128 units sold. The authors selected 20 machines across five site types for the simulated network. Their held-out validation reproduced total sales volume within 0.7 percent, with a weighted absolute percentage error of 0.473, compared with 0.606 for a naive reference. Product popularity and subcategory mix also improved over simpler alternatives.
Those validation results support the simulator’s demand behavior, but they do not make the environment a complete replica of vending operations. The authors state that price and environmental responses remain unvalidated by design. Margins, shelf lives, and several operating costs are benchmark settings rather than estimates established by the held-out sales test.
Agents versus a conventional inventory policy
The authors evaluated five LLM agents using identical prompts, tools, budgets, starting capital, and a 180-day horizon. They compared them with a deterministic rule-based inventory-control system using rolling demand estimates, order-up-to replenishment, weekly restocking, and local price adjustments.
Four of the five evaluated LLMs underperformed the rule-based baseline on terminal net assets. GPT-6.1 Sol was the exception, reaching a mean of 309.35 thousand CNY across three runs, while the table reports 210.97 thousand CNY for the rule-based reference. GLM-5.3-Flash averaged 148.43 thousand CNY, DeepSeek V4.1 Flash 133.92 thousand, Claude Sonnet 5.5 103.04 thousand, and Kimi K3 91.52 thousand.
The ranking changed when inference cost was included. GPT produced the greatest absolute return, but DeepSeek had the highest estimated net-asset growth per dollar of API expenditure. The authors report that GPT’s mean estimated API cost was approximately 80 times DeepSeek’s, while its net-asset growth was about 2.9 times as large. This is an important distinction for deployments where operating margin and model cost must be evaluated together.
Why strong metrics did not always mean strong businesses
AutoVend’s operational diagnostics show why a single score can conceal important weaknesses. GPT led the LLMs in assortment quality and profitability, but it also had the largest pricing regret among the evaluated systems. Claude achieved the lowest waste rate and pricing regret, yet its machines were empty on 25.21 percent of site-days. DeepSeek kept shelves available more often, but recorded the highest waste rate at 1.85 percent.
Coordination was also incomplete. At six checkpoints, the authors measured how much an agent could improve by reassigning its existing assortments to better-matched sites. GPT’s actual assignments were 7.05 percent below this restricted optimum, compared with 2.61 percent for the rule-based system. The finding does not show that GPT was unprofitable; rather, it shows that high aggregate returns can coexist with unrealized value in cross-site product placement.
Physical execution created another failure point. Warehouse inventory was not useful to customers until a paid visit transferred it to a machine. In one reported DeepSeek trajectory, a visit generated only 5.14 CNY in gross profit between visits while another visit cost 35 CNY. The example illustrates why replenishment frequency must be matched to local turnover instead of triggered by every small inventory change.
Feedback was not consistently converted into better policy
The authors found limited evidence of sustained improvement over the episode. Daily asset growth decreased between early and late comparison windows for all five model means and in 13 of 15 individual executions, even though some agents improved isolated measures such as waste or pricing.
Trajectory inspection identified recurring problems: infeasible allocations made without checking product-level availability, repeated purchases of products already accumulating in the warehouse, fragmented site visits, and corrections that were not retained as durable operating rules. These are observed failure cases, not estimates of how often every agent would fail.
What the results mean for agent builders
AutoVend suggests that long-horizon business agents should be evaluated on accumulated economics, not just tool-call success or short-term task completion. A useful evaluation should expose shared resources, delayed effects, inventory aging, execution costs, and model-inference costs at the same time.
For practical systems, the authors’ findings point toward explicit state verification before batch actions, product-level inventory accounting, visit consolidation, and persistent policy updates rather than one-off error recovery. Those are reasonable design implications, but the paper does not demonstrate that adding any particular safeguard will solve the problem.
The evidence also needs careful qualification. The comparison used three runs per LLM, one frozen world and the same random seed for demand, and a simulator whose assumptions do not cover every real-world response. The authors therefore frame broader testing across environments and controlled interventions as future work. AutoVend is best read as a demanding diagnostic benchmark: it shows how quickly apparently capable agents can lose economic value when planning, execution, and learning must remain coherent over time.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.