WeaveBench
A Long-Horizon Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
GraphCanon updated 3w · GitHub synced 3w
Decision brief
WeaveBench is designed for evaluating computer-use agents that integrate both GUI and CLI interactions in real-world scenarios across various work domains.
Good fit when
- Use WeaveBench if you need to assess agents capable of handling tasks that require intermingling graphical user interface operations with command-line or code-based actions.
- Opt for this tool when working on projects where performance evaluation must account for a hybrid interaction style, as opposed to focusing solely on GUI or CLI capabilities.
Avoid when
- Avoid WeaveBench if your testing needs do not involve scenarios that require the integration of both GUI and CLI operations.
- Do not use it when you are specifically interested only in benchmarking agents designed for single-channel tasks, either strictly CLI-based or purely graphical interface-driven.
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Very active (6d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- 16 low (16 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install WeaveBenchSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
WeaveBench benchmarks computer-use agents that integrate GUI and CLI interactions in real-world scenarios across various work domains.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Jul 29, 2026
- Languages
- python
Source: github.language+pyproject.toml · Jul 29, 2026
Categories
Tags
README
WeaveBench
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
A benchmark for computer-use agents that weave GUI and CLI/code together in real deployed runtimes. 114 long-horizon, real-world tasks across 8 work domains, each requiring the agent to interleave GUI observation with CLI/code execution in one trajectory, scored by a trajectory-aware Agent-as-Judge that reads the full chat trace + deliverables and zeroes fabricated evidence. The best frontier pairing clears just 41.2% — far from saturation.
📰 News
- 2026-07-08 🧵 Per-case agent trajectories released — browse the best hybrid GUI+CLI rollout for every one of the 114 tasks (thinking · actions · screenshots · deliverables · judge scores), bilingual viewer. Trajectory gallery
- 2026-06-30 🚀 OSWorld-V2 hybrid GUI+CLI harness — GPT-5.5 gains +5.5 pt Binary (13.0→18.5%) at ~2× tool-call efficiency.
experiments/osworld_v2_hybrid/ - 2026-06-12 🎉 Ranked #4 on HF Daily Papers with 104 upvotes.
- 2026-06-08 Initial preprint + project website live.
- 2026-06-06 🔬 OSWorld CLI-agent re-judge — an in-VM agent-as-judge recovers CLI successes the GUI grader misses (~2× more efficient than vision).
experiments/osworld_hybrid/ - 2026-06-05 🔥 WeaveBench released — 114 channel-non-substitutable tasks, 8 domains, trajectory-aware agentic judge; 9 backbones × 4 runtimes, best pairing 41.2% PassRate.
docs/REPRODUCE.md
Want your model on the board? See Submit your results.
TL;DR
What — 114 long-horizon tasks across 8 domains (WEB, DAV, OPS, DOC, DES, GAM, SPA, DSK), sourced from real user requests with traceable provenance.
The twist — Each task is channel-non-substitutable: no single-channel rewrite can solve it. GUI exposes transient rendered state (canvas, dialogs, charts); CLI/code carries persistent state (configs, logs, services). You need both, woven together.
Scoring — A trajectory-aware agentic judge re-fetches evidence over multiple turns and zeros any rollout with high-confidence fabrication (synthetic screenshots, hard-coded metrics). Outcome-only grading overestimates GPT-5.5 by +20 pts (53.5% → audited 33.3%).
Headline — Best model-runtime pairing (Claude Opus 4.7 + Claude Code) = 41.2% PassRate, vs >78% the same backbones reach on OSWorld-Verified.
🎬 Demo
https://github.com/weavebench/WeaveBench/raw/main/docs/media/rabbitmq_dlq_demo.mp4
An agent diagnosing a RabbitMQ dead-letter-queue routing black-hole end-to-end (OPS domain, sped up 10×). It cross-checks the broker over the CLI and the Management UI on screen, fixes the binding, and re-verifies — exactly the GUI↔CLI interleaving WeaveBench requires. Can't see the player? Watch the demo here.
For agents
This page has a .md twin and JSON over the API.