Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis
AIPR assessment
Problem difficulty: moderately competitive and increasingly crowded, since real-world robot benchmarking and VLA evaluation are active areas with many recent systems and benchmarks. The paper’s strengths reinforce each other, because the real hardware setting, public dataset release, and structured failure analysis make the benchmark more reusable than a simple success-rate comparison. The weaknesses also compound, because the contribution is mainly evaluative, so limited episode counts and miss
Abstract
Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored. We present a standardized real-world benchmark for evaluating representative VLA and imitation learning policies on the low-cost SO-101 robotic platform. The benchmark comprises four representative manipulation tasks together with unified evaluation protocols, enabling systematic comparison under embodiment uncertainty. Using real-world teleoperated demonstrations, we fine-tune and evaluate $π_{0.5}$, SmolVLA, Wall-X, and ACT directly on the physical platform. Beyond conventional task success rates, the benchmark incorporates a structured failure taxonomy, semantic- and execution-level failure decomposition, and recovery-aware evaluation metrics to characterize policy robustness. Experimental results show that stronger pretrained VLA policies generally outperform the imitation learning baseline, although performance remains highly task-dependent under low-cost robotic deployment conditions. Execution instability emerges as the dominant failure source, while recovery capability varies substantially across architectures. These results highlight the importance of failure and recovery analysis beyond binary task success and establish SO-101 as a practical benchmark for evaluating embodied AI systems under realistic low-cost robotic deployment conditions.
Score Breakdown
More from this week
More in Robotics
- Learning High-Frequency Continuous Action Chunks in Latent Space
- Dynamic Neural Koopman Distillation for Real-Time Robot Control Using Diffusion Models
- AcroRL: Learning Aggressive Quadrotor Inversion using Bidirectional Thrust
- 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
- AERO-VIS: Asynchronous Event-based Real-time Onboard Visual-Inertial SLAM