Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
As large language models (LLMs) evolve to understand and interact with continuous audiovisual data streams, evaluating their ability to assist users proactively and reliably remains a challenge. The recent work introducing Omni-InteractScope addresses this gap by providing a comprehensive benchmark that tests how well multimodal LLMs connect streaming evidence to timely and sustained user interventions.
The Problem: Evaluating Proactive Assistance in Streaming Contexts
Omnimodal large language models (Omni-LLMs) like GPT-4o and Qwen-Omni variants are designed to process joint video and audio inputs and assist users in real time. Unlike static video understanding tasks, these models must decide when to respond, what to respond with, and whether to invoke external tools or provide continuous guidance. Existing benchmarks primarily focus on comprehension or isolated interactions, offering limited insight into the models’ ability to deliver timely, context-aware, and sustained assistance.
Key challenges include:
- Timely tool invocation: Calling the right tool at the right moment based on detected events.
- Query-free assistance: Providing reminders or alerts without explicit user queries.
- Coverage of continuing interactions: Sustaining multi-step guidance or reminders as situations evolve.
Method: Omni-InteractScope Benchmark Design
The authors developed Omni-InteractScope, a benchmark comprising 1,479 human-annotated interactions with 1,790 temporal targets across eight task groups. These tasks span a broad spectrum of real-world assistance scenarios, including:
- Tool invocation triggered by specific events (e.g., image search when a person appears).
- Event notifications and risk/error warnings (e.g., alerting unsafe behavior).
- Cumulative state tracking and temporal conditions (e.g., reminding after a timer).
- Nonverbal/workflow cues (e.g., responding to gestures).
- Continuous responses and impairment assistance for visual or hearing support.
The benchmark links four dimensions: input modalities (video, audio, text, images), event triggers, intervention types, and user needs. Each target specifies both the required response content and a valid time window for that response, enabling evaluation of both what and when the model should act.
Importantly, some interactions include multiple successive targets, reflecting continuing obligations rather than one-off responses.
Experimental Setup and Models Evaluated
The authors evaluated 12 multimodal models spanning three categories:
- Proprietary prompted models: Gemini-3.1-Pro, Gemini-3.5-Flash, Qwen3.5-Omni-Plus.
- Open-source prompted multimodal LLMs: Qwen3-Omni, Qwen2.5-Omni, Qwen3-VL, InternVL3.5.
- Streaming models with stateful adapters: LiveCC, MiniCPM-o 4.5, MMDuet2, MOSS-VL, AURA.
Models received causal inputs at query time and subsequent timestamps, with evaluations distinguishing between offline probing (independent prefix calls) and online rolling (stateful streaming) modes.
Key Findings and Evidence
1. Timely Tool Invocation Is Unreliable
Across 200 tool-invocation targets, no evaluated model produced an on-time first response for more than 40% of cases. Joint accuracy combining correct tool choice and timing peaked at 32.0% (Qwen3.5-Omni-Plus). Notably, a control experiment showed that removing scene inputs did not significantly reduce Gemini-3.5-Flash’s tool invocation accuracy, indicating that models often rely heavily on request priors rather than grounding actions in observed events.
2. Query-Free Reminders Are Poorly Covered
For 122 query-free reminder targets (assistance without explicit questions), the best model reached only 23.0% timely recall, substantially lower than coverage on explicit audiovisual questions (up to 62.3%). Experiments manipulating response history and cooldown policies revealed that response frequency and scheduling strongly influence reminder coverage, underscoring the importance of response policy design.
3. Limited Coverage of Complete Guidance Sequences
Among 26 multi-step guidance sequences, nine models failed to cover any sequence fully, and the best model completed only three sequences. While step-level recall reached 51.7%, complete coverage—which requires delivering every step in order—remains a significant challenge. Statistical analysis suggested that incomplete guidance coverage aligns with independent step-level success rates rather than indicating distinct memory or planning failures.
Diagnostic Insights
- Scene Evidence vs. Request Priors: Tool invocation success often stems from recognizing the requested tool in text rather than detecting the event in the audiovisual stream. Scene ablation experiments confirmed this, revealing that joint timing-and-tool accuracy alone does not prove event grounding.
- Response Policies Affect Coverage: Incorporating response history reduces output volume without harming recall, while adding cooldowns reduces both output and coverage. This highlights the trade-off between responsiveness and verbosity in streaming assistants.
- Event Guidance and Scheduling: Providing explicit event cues and scheduling response opportunities can improve coverage, but benefits vary by model and are not guaranteed under autonomous response conditions.
Limitations and Considerations
The benchmark focuses on evaluating response content and timing but does not assess downstream execution of tool calls or closed-loop task success. The reliance on human-annotated reference responses and GPT-4o-based content judges introduces potential subjectivity, although rigorous cross-review and consistency checks were performed.
Additionally, the benchmark’s temporal tolerance windows (typically ±2 seconds) affect scoring sensitivity, and some models’ response policies and input modalities differ, complicating direct comparisons.
Practical Implications
Omni-InteractScope provides a valuable resource for developing and benchmarking omnimodal assistants capable of proactive, agentic interaction in continuous streams. The findings emphasize the need for:
- Improved event grounding: Models must better integrate scene evidence rather than relying on request priors.
- Enhanced timing mechanisms: Accurate detection of when to act is critical for usefulness.
- Robust sustained assistance: Supporting multi-step guidance and query-free reminders requires models to maintain state and plan over time.
- Careful response policy design: Balancing responsiveness with output volume is essential to avoid overwhelming or missing user needs.
By highlighting current limitations, the benchmark guides future research toward more reliable and context-aware streaming assistants that can meet diverse user needs in real-world scenarios.
“Evaluation across 12 models reveals substantial limitations in both timely intervention and sustained assistance… These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.” — Authors
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.