Dynamic Neural Koopman Distillation for Real-Time Robot Control Using Diffusion Models

AIPR assessment

This is a hard, competitive problem space, real-time robot control with diffusion policies is crowded, latency-sensitive, and baseline-heavy, so even incremental gains matter more than in a niche task. The strengths reinforce one another: the method is simple enough to deploy, the latency reduction is large, and the hardware result is meaningful. The main weaknesses also compound: the method has multiple moving parts, but the paper does not show detailed ablations in the provided text, and the s

Abstract

Diffusion models excel at generating diverse and multimodal trajectories for robotic planning, yet their iterative denoising process introduces latency that is incompatible with high-frequency closed-loop control. To address this problem, we propose Dynamic Neural Koopman Distillation, a framework that distills multistep diffusion inference into a single forward pass while retaining the multimodal expressivity of the teacher model. Specifically, we introduce a Factorized Dynamic Koopman layer that models the denoising process through a factorized latent transition with state-dependent modal gains. We evaluate the proposed method on standard D4RL MuJoCo locomotion benchmarks and a physical Kinova manipulator, comparing against one-step baselines. The results show that our method significantly outperforms existing one-step distillation approaches on the reported locomotion tasks, and reduces the inference latency to the millisecond regime compared with the teacher policy. Hardware experiments further demonstrate that our method enables smooth and fast closed-loop execution while maintaining task success and comparable accuracy. A project page is available at https://fdkoopman.github.io/.

Score Breakdown

More from this week

• Optimus: Elastic Decoding for Efficient Diffusion LLM Serving
• TubiFM: Unified Item, Carousel, and Search Ranking for Streaming Discovery
• Context Features Are Cheap: Rank-Aware Decomposition for Efficient Feature Interaction in Recommender Systems
• Rethinking Continual Anomaly Detection on the Edge: Benchmarking Under Realistic Industrial Conditions
• Learning High-Frequency Continuous Action Chunks in Latent Space

More in Robotics

• Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis
• vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
• Learning High-Frequency Continuous Action Chunks in Latent Space
• AcroRL: Learning Aggressive Quadrotor Inversion using Bidirectional Thrust
• 123D: Unifying Multi-Modal Autonomous Driving Data at Scale