Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows

AIPR assessment

Problem difficulty is high and crowded: agent benchmark design for evolving workflow demand is a competitive area with many recent systems, and claims of improvement are harder to make convincing here than in a niche task. The strongest aspects reinforce each other well, since the live signal sourcing, controlled execution, action-grounded grading, and public leaderboard all point to the same benchmark story. The main weaknesses also compound, because the demand side is still a proxy, the live-r

Abstract

LLM agents are expected to complete end-to-end units of work across software tools, business services, and local workspaces. Yet many agent benchmarks freeze a curated task set at release time and grade mainly the final response, making it difficult to evaluate agents against evolving workflow demand or verify whether a task was executed. We introduce Claw-Eval-Live, a live benchmark for workflow agents that separates a refreshable signal layer, updated across releases from public workflow-demand signals, from a reproducible, time-stamped release snapshot. Each release is constructed from public workflow-demand signals, with ClawHub Top-500 skills used in the current release, and materialized as controlled tasks with fixed fixtures, services, workspaces, and graders. For grading, Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts, using deterministic checks when evidence is sufficient and structured LLM judging only for semantic dimensions. The release contains 105 tasks spanning controlled business services and local workspace repair, and evaluates 13 frontier models under a shared public pass rule. Experiments reveal that reliable workflow automation remains far from solved: the leading model passes only 66.7% of tasks and no model reaches 70%. Failures are structured by task family and execution surface, with HR, management, and multi-system business workflows as persistent bottlenecks and local workspace repair comparatively easier but unsaturated. Leaderboard rank alone is insufficient because models with similar pass rates can diverge in overall completion, and task-level discrimination concentrates in a middle band of tasks. Claw-Eval-Live suggests that workflow-agent evaluation should be grounded twice, in fresh external demand and in verifiable agent action.

Score Breakdown

More from this week

• Beyond Visual Fidelity: Benchmarking Super-Resolution Models for Large-Scale Remote Sensing Imagery via Downstream Task Integration
• Eliminating Hidden Serialization in Multi-Node Megakernel Communication
• Succinct Graph Representations and Algorithmic Applications
• A Faster Deterministic Algorithm for Fully Dynamic Maximal Matching
• Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs

More in AI

• GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines
• The CIFAR Synthetic Evidence Corpus for Detecting AI-Generated Evidence
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
• SCOPE: Cost-Efficient Model Selection for Compound AI Systems under Quality Constraints
• Context Features Are Cheap: Rank-Aware Decomposition for Efficient Feature Interaction in Recommender Systems