{"data":{"slug":"weavebench-weavebench","name":"WeaveBench","tagline":"WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces","github_url":"https://github.com/weavebench/WeaveBench","owner":"weavebench","repo":"WeaveBench","owner_avatar_url":"https://avatars.githubusercontent.com/u/290423798?v=4","primary_language":"Python","stars":151,"forks":0,"topics":["agent-as-judge","benchmark","computer-use-agent","gui-agent","hybrid-interface","llm-evaluation"],"archived":false,"github_pushed_at":"2026-07-08T08:57:29+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/weavebench-weavebench","markdown_url":"https://www.graphcanon.com/tools/weavebench-weavebench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/weavebench-weavebench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=weavebench-weavebench","description":"WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces","homepage_url":"https://weavebench.github.io","license":"MIT","open_issues":3,"watchers":0,"ai_summary":null,"readme_excerpt":"# WeaveBench\n\n> **WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces**\n\n<p align=\"center\">\n<a href=\"https://arxiv.org/abs/2606.09426\"><img src=\"https://img.shields.io/badge/arXiv-2606.09426-b31b1b.svg?style=flat-square\" alt=\"arXiv\" /></a>\n<a href=\"https://weavebench.github.io\"><img src=\"https://img.shields.io/badge/🌐-Website-1f6feb.svg?style=flat-square\" alt=\"Website\" /></a>\n<a href=\"https://huggingface.co/datasets/wanlilll/WeaveBench\"><img src=\"https://img.shields.io/badge/🤗-Dataset-ffce00.svg?style=flat-square\" alt=\"Dataset\" /></a>\n<a href=\"https://huggingface.co/papers/2606.09426\"><img src=\"https://img.shields.io/badge/🤗_Daily_Papers-%234-ff8800.svg?style=flat-square\" alt=\"Daily Papers\" /></a>\n<a href=\"./LICENSE\"><img src=\"https://img.shields.io/badge/License-MIT-2ea44f.svg?style=flat-square\" alt=\"License\" /></a>\n</p>\n\n**A benchmark for computer-use agents that weave GUI and CLI/code together in real deployed runtimes.** 114 long-horizon, real-world tasks across 8 work domains, each requiring the agent to **interleave GUI observation with CLI/code execution in one trajectory**, scored by a **trajectory-aware Agent-as-Judge** that reads the full chat trace + deliverables and zeroes fabricated evidence. The best frontier pairing clears just **41.2%** — far from saturation.\n\n## 📰 News\n\n- **2026-07-08** 🧵 **Per-case agent trajectories released** — browse the best hybrid GUI+CLI rollout for every one of the 114 tasks (thinking · actions · screenshots · deliverables · judge scores), bilingual viewer. [Trajectory gallery](https://weavebench.github.io/trajectories/)\n- **2026-06-30** 🚀 **OSWorld-V2 hybrid GUI+CLI harness** — GPT-5.5 gains **+5.5 pt** Binary (13.0→18.5%) at ~2× tool-call efficiency. [`experiments/osworld_v2_hybrid/`](./experiments/osworld_v2_hybrid)\n- **2026-06-12** 🎉 Ranked **#4 on [HF Daily Papers](https://huggingface.co/papers/2606.09426)** with **104 upvotes**.\n- **2026-06-08** Initial preprint + [project website](https://weavebench.github.io) live.\n- **2026-06-06** 🔬 **OSWorld CLI-agent re-judge** — an in-VM agent-as-judge recovers CLI successes the GUI grader misses (~2× more efficient than vision). [`experiments/osworld_hybrid/`](./experiments/osworld_hybrid)\n- **2026-06-05** 🔥 **WeaveBench released** — 114 channel-non-substitutable tasks, 8 domains, trajectory-aware agentic judge; 9 backbones × 4 runtimes, best pairing **41.2% PassRate**. [`docs/REPRODUCE.md`](./docs/REPRODUCE.md)\n\n<sub>Want your model on the board? See [Submit your results](#-submit-your-results).</sub>\n\n## TL;DR\n\n**What** — 114 long-horizon tasks across 8 domains (WEB, DAV, OPS, DOC, DES, GAM, SPA, DSK), sourced from real user requests with traceable provenance.\n\n**The twist** — Each task is *channel-non-substitutable*: no single-channel rewrite can solve it. GUI exposes transient rendered state (canvas, dialogs, charts); CLI/code carries persistent state (configs, logs, services). You need both, woven together.\n\n**Scoring** — A trajectory-aware agentic judge re-fetches evidence over multiple turns and **zeros any rollout with high-confidence fabrication** (synthetic screenshots, hard-coded metrics). Outcome-only grading overestimates GPT-5.5 by **+20 pts** (53.5% → audited 33.3%).\n\n**Headline** — Best model-runtime pairing (Claude Opus 4.7 + Claude Code) = **41.2% PassRate**, vs >78% the same backbones reach on OSWorld-Verified.\n\n## 🎬 Demo\n\nhttps://github.com/weavebench/WeaveBench/raw/main/docs/media/rabbitmq_dlq_demo.mp4\n\n<p align=\"center\">\n  <em>An agent diagnosing a RabbitMQ dead-letter-queue routing black-hole end-to-end (OPS domain, sped up 10×). It cross-checks the broker over the CLI and the Management UI on screen, fixes the binding, and re-verifies — exactly the GUI↔CLI interleaving WeaveBench requires. Can't see the player? <a href=\"https://weavebench.github.io/static/videos/rabbitmq_dlq_topology_mgmt.mp4\">Watch the demo here</a>.</em>\n</p>\n\n<br />\n\n<p align=\"center\">\n  <img","github_created_at":"2026-06-05T16:21:28+00:00","created_at":"2026-07-11T12:02:04.234115+00:00","updated_at":"2026-07-11T12:02:26.073161+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"agent-as-judge","name":"agent-as-judge"},{"slug":"benchmark","name":"benchmark"},{"slug":"computer-use-agent","name":"computer-use-agent"},{"slug":"gui-agent","name":"gui-agent"},{"slug":"hybrid-interface","name":"hybrid-interface"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":1260566992,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T12:02:05.004Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":1,"days_since_push":3,"last_release_at":"2026-06-05T16:22:56Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":16,"high_count":0,"last_scan_at":"2026-07-11T12:02:06.344Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T12:02:06.102Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T12:02:06.102Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T12:02:06.102Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T12:02:06.102Z"}}}}