When Should a Streaming Video LLM Speak? Evidence‑Condition Alignment via Explicit Scene Graphs (Response‑G1)

With the rise of multimodal large language models, interaction between humans and AI is shifting from a simple command‑execute pattern to a truly symbiotic relationship, where agents act as proactive intelligences that continuously perceive the environment and decide when to intervene.

The core difficulty of proactive streaming video understanding is not only recognizing visual content but also determining whether the accumulated visual evidence satisfies the response condition embedded in the user query . Implicit representations cause models to wobble between similar frames, sometimes answering when they should stay silent.

To address this, researchers from Northwestern Polytechnical University, Hong Kong University of Science and Technology, and Tsinghua University presented Response‑G1 at ACL 2026. The framework explicitly models evidence and condition alignment through a unified scene‑graph representation, combines dynamic memory retrieval, and employs trigger‑based prompting, all without fine‑tuning the underlying video‑LLM.

Framework overview :

Online query‑guided scene‑graph generation: For each streaming video segment centered on the current time step, the model outputs scene‑graph nodes (objects and attributes) and edges (relations) as triples. The generation prompt incorporates the user query, encouraging the video‑LLM to extract query‑relevant sub‑structures and produce a focused graph.

Memory‑augmented scene‑graph retrieval: A growing memory bank stores scene graphs from previous time steps. During retrieval, each graph’s triples are linearized into natural‑language phrases and concatenated with the parsed response‑condition graph from the query. Both are embedded by the same text encoder, pooled, and compared via cosine similarity; the top‑K graphs become the evidence context.

Retrieval‑enhanced streaming trigger and answer: At each decision step, the model receives video frame embeddings, timestamp‑prefixed retrieved scene‑graph encodings, and a trigger instruction (e.g., “Should you answer now? Answer Yes/No only”). If the trigger predicts silence, observation continues; otherwise, the original question is appended to the context and a natural‑language answer is generated.

Experiments were conducted on the OVO‑Bench and StreamingBench benchmarks using Qwen3‑VL‑8B as the backbone and following standard input resolution and frame‑sampling settings. Results show that on the active sub‑tasks, Response‑G1 improves performance by 12.8 % on OVO‑Bench and 15.1 % on StreamingBench’s PO task, while also delivering stable gains on passive settings.

Ablation studies reveal that (1) scene‑graph‑based retrieval consistently boosts both active and passive performance, and timestamped graph encoding further enhances evidence understanding; (2) query‑guided graph generation outperforms target‑guided generation, which can produce spurious triples and cause premature responses.

A visual case illustrates a user query “What did the boy in the red T‑shirt do after leaving?” The system correctly retrieves the relevant scene graph at timestamp 18:51 and triggers a response, whereas baseline methods remain silent throughout the video.

In conclusion, the study demonstrates that making the timing decision an explicit, interpretable graph‑alignment problem not only solves the “when to speak” issue but also improves answer accuracy for temporally‑aware questions. This structured intermediate representation offers a composable foundation for future multimodal assistants, long‑stream memory, and more sophisticated human‑machine collaboration.