WeaveBench

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

A benchmark for computer-use agents that weave GUI and CLI/code together in real deployed runtimes. 114 long-horizon, real-world tasks across 8 work domains, each requiring the agent to interleave GUI observation with CLI/code execution in one trajectory, scored by a trajectory-aware Agent-as-Judge that reads the full chat trace + deliverables and zeroes fabricated evidence. The best frontier pairing clears just 41.2% — far from saturation.

📰 News

• 2026-07-08 🧵 Per-case agent trajectories released — browse the best hybrid GUI+CLI rollout for every one of the 114 tasks (thinking · actions · screenshots · deliverables · judge scores), bilingual viewer. Trajectory gallery
• 2026-06-30 🚀 OSWorld-V2 hybrid GUI+CLI harness — GPT-5.5 gains +5.5 pt Binary (13.0→18.5%) at ~2× tool-call efficiency. experiments/osworld_v2_hybrid/
• 2026-06-12 🎉 Ranked #4 on HF Daily Papers with 104 upvotes.
• 2026-06-08 Initial preprint + project website live.
• 2026-06-06 🔬 OSWorld CLI-agent re-judge — an in-VM agent-as-judge recovers CLI successes the GUI grader misses (~2× more efficient than vision). experiments/osworld_hybrid/
• 2026-06-05 🔥 WeaveBench released — 114 channel-non-substitutable tasks, 8 domains, trajectory-aware agentic judge; 9 backbones × 4 runtimes, best pairing 41.2% PassRate. docs/REPRODUCE.md

_{Want your model on the board? See Submit your results.}

TL;DR

What — 114 long-horizon tasks across 8 domains (WEB, DAV, OPS, DOC, DES, GAM, SPA, DSK), sourced from real user requests with traceable provenance.

The twist — Each task is channel-non-substitutable: no single-channel rewrite can solve it. GUI exposes transient rendered state (canvas, dialogs, charts); CLI/code carries persistent state (configs, logs, services). You need both, woven together.

Scoring — A trajectory-aware agentic judge re-fetches evidence over multiple turns and zeros any rollout with high-confidence fabrication (synthetic screenshots, hard-coded metrics). Outcome-only grading overestimates GPT-5.5 by +20 pts (53.5% → audited 33.3%).

Headline — Best model-runtime pairing (Claude Opus 4.7 + Claude Code) = 41.2% PassRate, vs >78% the same backbones reach on OSWorld-Verified.

🎬 Demo

https://github.com/weavebench/WeaveBench/raw/main/docs/media/rabbitmq_dlq_demo.mp4

An agent diagnosing a RabbitMQ dead-letter-queue routing black-hole end-to-end (OPS domain, sped up 10×). It cross-checks the broker over the CLI and the Management UI on screen, fixes the binding, and re-verifies — exactly the GUI↔CLI interleaving WeaveBench requires. Can't see the player? Watch the demo here.