ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving

AIPR assessment

This is a hard, competitive systems problem, not a niche benchmark: MoE serving, expert placement, and accelerator variability have all been actively optimized by multiple groups. The strengths reinforce each other, the conceptual insight is simple, the evidence is real-hardware, and the gains are consistent across several conditions, which makes the result feel credible and useful. The weaknesses also compound, though less severely: the method relies on offline profiling plus online migration,

Abstract

In distributed Mixture-of-Experts (MoE) inference, input-dependent token routing interacts with GPU performance variability to create persistent stragglers under synchronized execution, where the slowest GPU determines layer latency. This performance variability is inherent to modern accelerators: manufacturing variation, power limits, and thermal conditions introduce measurable execution-time differences across nominally identical GPUs. The core challenge is that MoE execution-time imbalance arises from the interaction of workload skew and hardware asymmetry. Token routing produces uneven and layer-varying expert loads, while GPU throughput depends on device-specific operating characteristics and workload intensity. Prior work mitigates routing skew but assumes homogeneous hardware, optimizing token balance rather than execution latency. As a result, even balanced token assignments can leave hardware-induced stragglers unaddressed. Thus, we propose Variability-Informed Binning of Experts (ViBE), a hardware-aware expert placement framework that minimizes execution-time imbalance across GPUs. ViBE combines per-GPU performance modeling with expert activation profiling to assign high-load experts to faster devices and low-load experts to slower ones, reducing layer-level stragglers without modifying model semantics or hardware. Because both workload characteristics and effective GPU throughput can shift across serving conditions, ViBE supports lightweight recalibration under workload/performance drift to refresh its routing and performance estimates when needed. Results show that ViBE consistently reduces execution-time imbalance and improves SLO attainment by 14%, while lowering P90 TTFT by up to 45%. We further show that the impact of hardware variability increases at scale, making variability-aware placement important for efficient, high-utilization LLM serving.

Score Breakdown

More from this week

• Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
• On Thin Perfect Matchings up to Polylogarithmic Factors
• LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
• HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces

More in Computer Architecture

• ScaleDisturb: Exploiting Temporal Asymmetry to Amplify Read Disturbance in Modern DRAM Chips
• Optimus: Elastic Decoding for Efficient Diffusion LLM Serving
• DORA: Dataflow-Instruction Orchestration Architecture for DNN Acceleration
• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
• Loaded Dice: Solving the Non-Selection Problem for Scalable Probabilistic RowHammer Defense