---
title: "WeaveBench"
type: "tool"
slug: "weavebench-weavebench"
canonical_url: "https://www.graphcanon.com/tools/weavebench-weavebench"
github_url: "https://github.com/weavebench/WeaveBench"
homepage_url: "https://weavebench.github.io"
stars: 151
forks: 0
primary_language: "Python"
license: "MIT"
archived: false
categories: ["ai-agents", "llm-frameworks", "vector-databases"]
tags: ["agent-as-judge", "benchmark", "computer-use-agent", "gui-agent", "hybrid-interface", "llm-evaluation", "python"]
updated_at: "2026-07-11T12:02:26.073161+00:00"
---

# WeaveBench

> WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

## Facts

- Repository: https://github.com/weavebench/WeaveBench
- Homepage: https://weavebench.github.io
- Stars: 151 · Forks: 0 · Open issues: 3 · Watchers: 0
- Primary language: Python
- License: MIT
- Last pushed: 2026-07-08T08:57:29+00:00

## Trust & health

_Signals computed from public GitHub metadata. Not a security guarantee._

- Maintenance: Very active (computed 2026-07-11T12:02:05.004Z)
- Security scan: Findings present (0 critical, 0 high, 0 medium, 16 low) · last scan 2026-07-11T12:02:06.344Z
- Full report: [trust report](/tools/weavebench-weavebench/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/weavebench-weavebench/trust)

## Categories

- [AI Agents](/categories/ai-agents.md)
- [LLM Frameworks](/categories/llm-frameworks.md)
- [Vector Databases](/categories/vector-databases.md)

## Tags

agent-as-judge, benchmark, computer-use-agent, gui-agent, hybrid-interface, llm-evaluation, python

## Category neighbours (exploratory)

_Same-category tools for discovery only - not curated alternatives. Cap shown at six._

- [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active]
- [ECC](/tools/affaan-m-ecc.md) - The agent harness performance optimization system for AI agents (★ 228,395) [Very active]
- [hermes-agent](/tools/nousresearch-hermes-agent.md) - The agent that grows with you (★ 212,994) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]

_+ 2 more not listed._

## README (excerpt)

_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._

```text
# WeaveBench

> **WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces**

<p align="center">
<a href="https://arxiv.org/abs/2606.09426"><img src="https://img.shields.io/badge/arXiv-2606.09426-b31b1b.svg?style=flat-square" alt="arXiv" /></a>
<a href="https://weavebench.github.io"><img src="https://img.shields.io/badge/🌐-Website-1f6feb.svg?style=flat-square" alt="Website" /></a>
<a href="https://huggingface.co/datasets/wanlilll/WeaveBench"><img src="https://img.shields.io/badge/🤗-Dataset-ffce00.svg?style=flat-square" alt="Dataset" /></a>
<a href="https://huggingface.co/papers/2606.09426"><img src="https://img.shields.io/badge/🤗_Daily_Papers-%234-ff8800.svg?style=flat-square" alt="Daily Papers" /></a>
<a href="./LICENSE"><img src="https://img.shields.io/badge/License-MIT-2ea44f.svg?style=flat-square" alt="License" /></a>
</p>

**A benchmark for computer-use agents that weave GUI and CLI/code together in real deployed runtimes.** 114 long-horizon, real-world tasks across 8 work domains, each requiring the agent to **interleave GUI observation with CLI/code execution in one trajectory**, scored by a **trajectory-aware Agent-as-Judge** that reads the full chat trace + deliverables and zeroes fabricated evidence. The best frontier pairing clears just **41.2%** — far from saturation.

## 📰 News

- **2026-07-08** 🧵 **Per-case agent trajectories released** — browse the best hybrid GUI+CLI rollout for every one of the 114 tasks (thinking · actions · screenshots · deliverables · judge scores), bilingual viewer. [Trajectory gallery](https://weavebench.github.io/trajectories/)
- **2026-06-30** 🚀 **OSWorld-V2 hybrid GUI+CLI harness** — GPT-5.5 gains **+5.5 pt** Binary (13.0→18.5%) at ~2× tool-call efficiency. [`experiments/osworld_v2_hybrid/`](./experiments/osworld_v2_hybrid)
- **2026-06-12** 🎉 Ranked **#4 on [HF Daily Papers](https://huggingface.co/papers/2606.09426)** with **104 upvotes**.
- **2026-06-08** Initial preprint + [project website](https://weavebench.github.io) live.
- **2026-06-06** 🔬 **OSWorld CLI-agent re-judge** — an in-VM agent-as-judge recovers CLI successes the GUI grader misses (~2× more efficient than vision). [`experiments/osworld_hybrid/`](./experiments/osworld_hybrid)
- **2026-06-05** 🔥 **WeaveBench released** — 114 channel-non-substitutable tasks, 8 domains, trajectory-aware agentic judge; 9 backbones × 4 runtimes, best pairing **41.2% PassRate**. [`docs/REPRODUCE.md`](./docs/REPRODUCE.md)

<sub>Want your model on the board? See [Submit your results](#-submit-your-results).</sub>

## TL;DR

**What** — 114 long-horizon tasks across 8 domains (WEB, DAV, OPS, DOC, DES, GAM, SPA, DSK), sourced from real user requests with traceable provenance.

**The twist** — Each task is *channel-non-substitutable*: no single-channel rewrite can solve it. GUI exposes transient rendered state (canvas, dialogs, charts); CLI/code carries persistent state (configs, logs, services). You need both, woven together.

**Scoring** — A trajectory-aware agentic judge re-fetches evidence over multiple turns and **zeros any rollout with high-confidence fabrication** (synthetic screenshots, hard-coded metrics). Outcome-only grading overestimates GPT-5.5 by **+20 pts** (53.5% → audited 33.3%).

**Headline** — Best model-runtime pairing (Claude Opus 4.7 + Claude Code) = **41.2% PassRate**, vs >78% the same backbones reach on OSWorld-Verified.

## 🎬 Demo

https://github.com/weavebench/WeaveBench/raw/main/docs/media/rabbitmq_dlq_demo.mp4

<p align="center">
  <em>An agent diagnosing a RabbitMQ dead-letter-queue routing black-hole end-to-end (OPS domain, sped up 10×). It cross-checks the broker over the CLI and the Management UI on screen, fixes the binding, and re-verifies — exactly the GUI↔CLI interleaving WeaveBench requires. Can't see the player? <a href="https://weavebench.github.io/static/videos/rabbitmq_dlq_topology_mgmt.mp4">Watch the demo here</a>.</em>
</p>

<br />

<p align="center">
  <img
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/weavebench-weavebench`](/api/graphcanon/tools/weavebench-weavebench)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_