DynMuon: A Dynamic Spectral Shaping View of Muon

AIPR assessment

Problem difficulty is high: this sits in a crowded, well-optimized area, large-scale LLM pretraining optimizers, where small gains matter and baselines are strong. The strengths reinforce each other well, a simple method, a plausible mechanism, broad ablations, and low overhead all point in the same direction. The main weaknesses also interact, the mechanistic model is intentionally idealized, and there is no external code or deployment evidence in the text, so the practical claim is supported m

Abstract

In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standard gradient descent methods, is to replace the usual update matrix $M=UΣV^\top$ with its polar factor $UV^\top$. In this work, we consider a class of Muon-like updates, where we replace the update $M$ with $UΣ^p V^\top$ for some parameter $p$. We call this a "spectral-shaping" operation, and develop a theory of how to pick $p$ which depends on (a) local curvature of the loss function, (b) noise stemming from stochastic gradients and label noise, and (c) training stage. Our theory and experimentation reveal a previously overlooked behavior: positive $p$ helps early by emphasizing high-curvature directions and accelerating signal contraction, while mildly negative $p$ helps later by reallocating update strength toward low-curvature directions that still contain useful training signals. Building on the insight, we propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training. Extensive experiments across model sizes, architectures, and training settings show that DynMuon consistently achieves lower validation loss than Muon, while requiring 10.6-26.5% fewer steps to reach the same target loss. Our code is available at https://github.com/fzwark/DynMuon.

Score Breakdown

More from this week

• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
• TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
• ContraFix: Agentic Vulnerability Repair via Differential Runtime Evidence and Skill Reuse
• Loaded Dice: Solving the Non-Selection Problem for Scalable Probabilistic RowHammer Defense
• Stop Starving or Stuffing Me: Boosting Firmware Fuzzing Efficiency with On-demand Input Delivery

More in Machine Learning

• STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
• Verifiable and Confidential DNN Inference on Low-End Edge Devices
• HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
• LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems
• TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks