{"data":{"slug":"weavebench-weavebench","name":"WeaveBench","tagline":"A Long-Horizon Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces","github_url":"https://github.com/weavebench/WeaveBench","owner":"weavebench","repo":"WeaveBench","owner_avatar_url":"https://avatars.githubusercontent.com/u/290423798?v=4","primary_language":"Python","stars":167,"forks":4,"topics":["agent-as-judge","benchmark","computer-use-agent","gui-agent","hybrid-interface","llm-evaluation"],"archived":false,"github_pushed_at":"2026-08-22T09:03:53+00:00","maintenance_label":"Active","stars_delta_30d":10,"url":"https://www.graphcanon.com/tools/weavebench-weavebench","markdown_url":"https://www.graphcanon.com/tools/weavebench-weavebench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/weavebench-weavebench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=weavebench-weavebench","description":"[EMNLP 2026] WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces","homepage_url":"https://weavebench.github.io","license":"MIT","open_issues":5,"watchers":0,"ai_summary":"WeaveBench benchmarks computer-use agents that integrate GUI and CLI interactions in real-world scenarios across various work domains.","readme_excerpt":"# WeaveBench\n\n> **WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces**\n\n<p align=\"center\">\n<a href=\"https://2026.emnlp.org/\"><img src=\"https://img.shields.io/badge/EMNLP_2026-Main_Conference-b5179e.svg?style=flat-square\" alt=\"EMNLP 2026 Main Conference\" /></a>\n<a href=\"https://arxiv.org/abs/2606.09426\"><img src=\"https://img.shields.io/badge/arXiv-2606.09426-b31b1b.svg?style=flat-square\" alt=\"arXiv\" /></a>\n<a href=\"https://weavebench.github.io\"><img src=\"https://img.shields.io/badge/🌐-Website-1f6feb.svg?style=flat-square\" alt=\"Website\" /></a>\n<a href=\"https://huggingface.co/datasets/wanlilll/WeaveBench\"><img src=\"https://img.shields.io/badge/🤗-Dataset-ffce00.svg?style=flat-square\" alt=\"Dataset\" /></a>\n<a href=\"https://huggingface.co/papers/2606.09426\"><img src=\"https://img.shields.io/badge/🤗_Daily_Papers-%234-ff8800.svg?style=flat-square\" alt=\"Daily Papers\" /></a>\n<a href=\"./LICENSE\"><img src=\"https://img.shields.io/badge/License-MIT-2ea44f.svg?style=flat-square\" alt=\"License\" /></a>\n</p>\n\n**A benchmark for computer-use agents that weave GUI and CLI/code together in real deployed runtimes.** 114 long-horizon, real-world tasks across 8 work domains, each requiring the agent to **interleave GUI observation with CLI/code execution in one trajectory**, scored by a **trajectory-aware Agent-as-Judge** that reads the full chat trace + deliverables and zeroes fabricated evidence. The best frontier pairing clears just **41.2%** — far from saturation.\n\n## 📰 News\n\n- **2026-08-22** 🎊 **Accepted to EMNLP 2026 Main Conference!** See you in Budapest 🇭🇺\n- **2026-07-08** 🧵 **Per-case agent trajectories released** — browse the best hybrid GUI+CLI rollout for every one of the 114 tasks (thinking · actions · screenshots · deliverables · judge scores), bilingual viewer. [Trajectory gallery](https://weavebench.github.io/trajectories/)\n- **2026-06-30** 🚀 **OSWorld-V2 hybrid GUI+CLI harness** — GPT-5.5 gains **+5.5 pt** Binary (13.0→18.5%) at ~2× tool-call efficiency. [`experiments/osworld_v2_hybrid/`](./experiments/osworld_v2_hybrid)\n- **2026-06-12** 🎉 Ranked **#4 on [HF Daily Papers](https://huggingface.co/papers/2606.09426)** with **104 upvotes**.\n- **2026-06-08** Initial preprint + [project website](https://weavebench.github.io) live.\n- **2026-06-06** 🔬 **OSWorld CLI-agent re-judge** — an in-VM agent-as-judge recovers CLI successes the GUI grader misses (~2× more efficient than vision). [`experiments/osworld_hybrid/`](./experiments/osworld_hybrid)\n- **2026-06-05** 🔥 **WeaveBench released** — 114 channel-non-substitutable tasks, 8 domains, trajectory-aware agentic judge; 9 backbones × 4 runtimes, best pairing **41.2% PassRate**. [`docs/REPRODUCE.md`](./docs/REPRODUCE.md)\n\n<sub>Want your model on the board? See [Submit your results](#-submit-your-results).</sub>\n\n## TL;DR\n\n**What** — 114 long-horizon tasks across 8 domains (WEB, DAV, OPS, DOC, DES, GAM, SPA, DSK), sourced from real user requests with traceable provenance.\n\n**The twist** — Each task is *channel-non-substitutable*: no single-channel rewrite can solve it. GUI exposes transient rendered state (canvas, dialogs, charts); CLI/code carries persistent state (configs, logs, services). You need both, woven together.\n\n**Scoring** — A trajectory-aware agentic judge re-fetches evidence over multiple turns and **zeros any rollout with high-confidence fabrication** (synthetic screenshots, hard-coded metrics). Outcome-only grading overestimates GPT-5.5 by **+20 pts** (53.5% → audited 33.3%).\n\n**Headline** — Best model-runtime pairing (Claude Opus 4.7 + Claude Code) = **41.2% PassRate**, vs >78% the same backbones reach on OSWorld-Verified.\n\n## 🎬 Demo\n\nhttps://github.com/weavebench/WeaveBench/raw/main/docs/media/rabbitmq_dlq_demo.mp4\n\n<p align=\"center\">\n  <em>An agent diagnosing a RabbitMQ dead-letter-queue routing black-hole end-to-end (OPS domain, sped up 10×). It cross-checks the broker over the CLI and the Management UI on screen, fixes th","github_created_at":"2026-06-05T16:21:28+00:00","created_at":"2026-07-11T12:02:04.234115+00:00","updated_at":"2026-09-20T07:00:33.79241+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agent-as-judge","name":"agent-as-judge"},{"slug":"benchmark","name":"benchmark"},{"slug":"computer-use-agent","name":"computer-use-agent"},{"slug":"gui-agent","name":"gui-agent"},{"slug":"hybrid-interface","name":"hybrid-interface"},{"slug":"llm-evaluation","name":"llm-evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":1260566992,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T07:00:31.466Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":28,"last_release_at":"2026-06-05T16:22:56Z","stars_delta_30d":10,"open_issues_delta_30d":1},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":16,"high_count":0,"last_scan_at":"2026-07-11T12:02:06.344Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T07:00:32.494Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-20T07:00:32.494Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T07:00:32.494Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-20T07:00:32.494Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use WeaveBench if you need to assess agents capable of handling tasks that require intermingling graphical user interface operations with command-line or code-based actions.","Opt for this tool when working on projects where performance evaluation must account for a hybrid interaction style, as opposed to focusing solely on GUI or CLI capabilities."],"when_not_to_use":["Avoid WeaveBench if your testing needs do not involve scenarios that require the integration of both GUI and CLI operations.","Do not use it when you are specifically interested only in benchmarking agents designed for single-channel tasks, either strictly CLI-based or purely graphical interface-driven."],"source":"enrich:decision_facts","observed_at":"2026-07-15T08:45:59.464Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"WeaveBench is designed for evaluating computer-use agents that integrate both GUI and CLI interactions in real-world scenarios across various work domains."}]}}