StreamEval: Evaluating Interactive Streaming Video Generation Models

A practical system can be decomposed into an input event gateway, session state manager, multimodal perception layer, control planner, streaming generation workers, temporal memory, output synchronizer and evaluation/telemetry pipeline. The gateway timestamps and orders incoming audio, text and image events. The state manager maintains active edits, protected entities and pending operations. The planner converts natural-language instructions into typed operations with explicit target, scope and deadline. Generation workers produce incremental video or avatar frames. A state validator checks whether the requested state transition occurred and whether unrelated content drifted. A recovery controller can retry, roll back or ask for clarification when the request is ambiguous or not supported. This is an engineering interpretation inspired by the benchmark's requirements, not an architecture prescribed by the authors.

A production team can adapt the benchmark as a release gate. Define a session event schema containing timestamp, modality, target entity, requested operation, deadline, persistence rule and protected attributes. Store the expected state after each event. Record generated frames, audio, user inputs, model outputs, tool calls and wall-clock timestamps. Evaluate each event independently for action completion, then evaluate the session for state drift, temporal defects, identity consistency, audio-video synchronization and preservation. Run repeated sessions at multiple durations and report per-task score distributions, not a single blended score. Include interruption, overlapping instructions, reference switches, cancellation, recovery after model stalls and requests that exceed the model's supported control surface.

The reported scores describe the tested model interfaces, task roster, benchmark version and judge protocol; they are not universal rankings of all video systems. The roster contains different task configurations and input routes, so cross-task comparisons should not be interpreted as direct head-to-head capability tests. Some systems do not expose custom-avatar or reference controls. Automated visual judging can miss subtle defects, and model-assisted benchmark construction may introduce biases despite authored oracles and calibration. The benchmark also uses scripted sessions and a finite set of assets; real users produce more ambiguous, adversarial and unpredictable requests. The paper itself identifies future coverage opportunities, including broader identities and environments and more complex concurrent edit restoration.

StreamEval reframes streaming video generation as a control-and-state problem, not only a synthesis problem. Its 300 sessions and 37-model evaluation show that visual fidelity, identity, continuity, following and successful editing are separable capabilities. The low top scores on timed avatar following, world-generation following and reference-edit success make a strong case for better interaction-aware training and evaluation. As video systems become persistent and multimodal, reliable state transitions and preservation will be as important as photorealism.