AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding
Most video-language systems are evaluated as if the model either waits for more footage or answers the user. The authors of AskStream argue that this action space is incomplete. In realistic streaming video, the information needed to answer may never arrive in the current view, may depend on a user preference, or may be impossible to infer from an ambiguous question. A system that only waits can wait indefinitely; one that answers anyway risks hallucinating.
The paper proposes a third action: ask. The model should decide whether to keep watching, request missing information, or answer. That request might ask for a clearer view of an occluded object, a different camera angle, clarification about which person the user means, or an updated preference. AskStream is the authors’ benchmark for measuring this behavior.
What AskStream evaluates
AskStream treats the interaction as a partially observable decision problem. At each point, the model sees the video observed so far, the user’s query, and the previous interaction history. It chooses among WAIT, ASK, and ANSWER. A successful turn requires both a suitable request at an appropriate time and a correct answer after the missing information is supplied.
The benchmark covers 438 videos and 1,401 questions, with 2,714 manually annotated frame-level timing points. Its videos span nine domains and range from short clips to streams lasting nearly two hours. The tasks are organized around two sources of uncertainty:
- Video uncertainty: missing visual evidence, missing viewpoints, unfamiliar individual concepts, and unfamiliar personalized actions.
- User uncertainty: ambiguous questions, false premises, missing preferences, and preferences that may have become outdated.
This taxonomy matters because different gaps call for different requests. Asking for a camera change is not equivalent to asking the user to identify a person, and requesting a current preference is different from correcting a false description in the question.
A timing-aware evaluation
The authors introduce Ask-F1, a harmonic-mean-style metric that combines the quality and timing of the ASK action with the quality and timing of the final answer. An ask receives credit only when its content is correct and it falls inside the annotated ASK window. The answer is then evaluated after a valid ask, with earlier actions inside the acceptable window receiving more timing credit.
This design reflects a practical constraint: asking the right question too early can be as unhelpful as asking the wrong question. The benchmark distinguishes two timing patterns. Sometimes the model must watch until the relevant evidence appears before it can recognize the information gap. In other cases, such as an already ambiguous user query, it should ask immediately and then wait for the answer-triggering video evidence.
To test models under streaming conditions, the authors also build StreamHarness, a hierarchical memory setup. It retains recent frames densely while compressing older observations into progressively longer-term summaries. The reported experiments include multiple-choice and open-ended settings, although the open-ended setting is substantially harder.
What the experiments show
The paper’s central result is that proactive asking remains a major weakness. In the multiple-choice evaluation, the strongest offline model reported by the authors reaches an overall Ask-F1 of 52.18%. The best streaming proactive model reaches only 7.22%, while several streaming systems score zero on most subtasks. In the open-ended evaluation, the strongest reported score falls to 29.19%, and the best streaming result is 4.01%.
The gap is not simply about producing final answers. Across the stronger offline models, answer scores are generally higher than ask scores. The authors identify early ASK and wrong ASK as common failure modes: models often detect that more information is needed, but request it before the valid time window or ask for information that will not resolve the uncertainty. Their oracle analysis reinforces this diagnosis. Supplying the correct time to ask raises performance on one studied task from 37.17% to 58.30%, while revealing both when and what to ask produces a reported 43.83-point improvement.
Additional experiments suggest that relevant information, rather than extra footage alone, drives the benefit of asking. In an uncertainty study, accuracy rises from 42.3% with blurred evidence to 62.3% after the requested evidence is revealed, approaching 63.1% with fully clear input. The authors also find that contextual memory helps more for user uncertainty than for video uncertainty, and that allowing more than one ask can improve performance before gains begin to level off.
Limitations and interpretation
AskStream is a useful diagnostic benchmark, but it is still a controlled approximation of real interaction. Each turn allows at most one ASK in the main protocol, even though real conversations may require several rounds of clarification. The benchmark starts with an explicit user query, so it does not test a fully autonomous assistant deciding to surface uncertainty without being prompted. It also focuses on video understanding rather than embodied systems that can change the physical environment or coordinate perception with actions.
The benchmark’s timing windows and candidate requests are carefully annotated, which supports reproducibility but may not capture overlapping or gradually changing opportunities in open-world streams. The open-ended results also depend on an automated judge, although the authors report a human comparison on a sampled set. These constraints mean the scores should be read as evidence about current performance on this benchmark, not as a complete measure of real-world reliability.
Why the idea matters for practitioners
For developers of video assistants, the practical lesson is to treat information acquisition as a first-class capability. A production system should not equate uncertainty with permission to guess, nor assume that watching longer will solve every problem. It needs explicit policies for asking for a better view, clarifying a referent, checking a user’s current preference, and correcting inconsistent premises.
AskStream also suggests how such systems should be tested. Accuracy after an oracle supplies the missing information can hide the harder failure: recognizing what is missing and asking at the right moment. Evaluations should therefore score the request itself, its timing, and the downstream answer separately. The authors’ results show that improving this decision layer may be at least as important as improving visual question answering. That conclusion is demonstrated by the benchmark findings; whether the same ranking holds across broader autonomous or embodied applications remains an open question.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.