Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

AIPR assessment

This is a hard, crowded, fast-moving survey target, with many groups publishing concurrently across overlapping subareas. The paper’s strengths compound well: the taxonomy, benchmark tables, and maintained repository reinforce one another and make the survey practically useful. The weaknesses are mostly survey-specific, namely some subjective grouping choices and a few bibliographic inconsistencies, rather than flaws that undermine the main value of the article. There are no suspicious empirical

Abstract

With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI, autonomous driving, etc. Through explicitly incorporating user actions into world state transition, recent literature empowers world modeling with interactivity in an action-conditioned video or 3D generation paradigm, further enhancing controllability over world evolutions and facilitating users to freely traverse, manipulate, navigate, and personalize the state evolution. In this paper, we aim to systematically review recent research trends, technical developments, evaluation benchmarks, and also propose future potential directions in interactive world modeling. Specifically, we first summarize recent efforts and trends in terms of application scenarios, world state evolution, and scene modality. Afterwards, we delve into three crucial technical challenges, including action-conditioned controllability, long-horizon interactions and memory, and action-following responsiveness for real-time interactivity. Furthermore, we also thoroughly compare existing benchmarks and metrics in four specific application fields: open-world exploration, game engine, autonomous driving, and robotics. Finally, we discuss several promising future directions in achieving next-generation interactive world modeling. The corresponding repository is publicly available at: https://github.com/liujiuming123/Awesome-Interactive-World-Model.

Score Breakdown

More from this week

• On Thin Perfect Matchings up to Polylogarithmic Factors
• ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
• LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
• HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces

More in Computer Vision

• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents
• Rethinking Continual Anomaly Detection on the Edge: Benchmarking Under Realistic Industrial Conditions
• FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing
• Beyond Visual Fidelity: Benchmarking Super-Resolution Models for Large-Scale Remote Sensing Imagery via Downstream Task Integration
• Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain