{"data":{"slug":"tiger-ai-lab-clawbench","name":"ClawBench","tagline":"Open-source benchmark for browser AI agents on daily tasks","github_url":"https://github.com/TIGER-AI-Lab/ClawBench","owner":"TIGER-AI-Lab","repo":"ClawBench","owner_avatar_url":"https://avatars.githubusercontent.com/u/144196744?v=4","primary_language":"Python","stars":796,"forks":58,"topics":["agent-evaluation","agentic-ai","ai-agent-benchmark","ai-agents","benchmark","browser-agent","browser-automation","browser-use","chrome-agent","chrome-extension","computer-use","dataset","evaluation","everyday-tasks","llm","llm-evaluation","online-tasks","real-world-benchmark","web-agent","web-agents"],"archived":false,"github_pushed_at":"2026-09-20T05:45:35+00:00","maintenance_label":"Very active","stars_delta_30d":264,"url":"https://www.graphcanon.com/tools/tiger-ai-lab-clawbench","markdown_url":"https://www.graphcanon.com/tools/tiger-ai-lab-clawbench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/tiger-ai-lab-clawbench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=tiger-ai-lab-clawbench","description":"Open-source benchmark for browser AI agents on daily tasks.","homepage_url":"https://claw-bench.com","license":"Apache-2.0","open_issues":46,"watchers":13,"ai_summary":"ClawBench is an open-source benchmark framework tailored to evaluate browser-based AI agents in handling everyday online tasks.","readme_excerpt":"## <img src=\"assets/icons/rocket.svg\" width=\"24\" height=\"24\"> Quick start\n\n```bash\ngit clone https://github.com/TIGER-AI-Lab/ClawBench.git && cd ClawBench && ./run.sh\n```\n\n<sub><i>Clone → configure → run. &nbsp; Root uv package. &nbsp; Docker-isolated harnesses.</i></sub>\n\n**Driving a coding agent instead?** Point it at [`AGENTS.md`](AGENTS.md) and prompt away.\n\n---\n\n### Install\n\n```bash\nuv tool install clawbench-eval\n```\n\n`pipx install clawbench-eval` and `python -m pip install clawbench-eval` work too. The installed commands are `clawbench`, `clawbench-run`, `clawbench-batch`, `clawbench-rescore`, `clawbench-analyze`, `clawbench-reproduce`, and `clawbench-harbor-adapt`.\n\nFor more granular control — modifying the driver, the bundled test cases, or the container build — clone the repo and use the root `uv` package entrypoint instead:\n\n```bash\ngit clone https://github.com/TIGER-AI-Lab/ClawBench.git && cd ClawBench && ./run.sh\n```\n\n**Prerequisites:** [Python 3.11+](https://python.org), [uv](https://docs.astral.sh/uv/), and a container engine — [Docker](https://www.docker.com/) **or** [Podman](https://podman.io/). ClawBench auto-detects whichever is installed; force one with `export CONTAINER_ENGINE=docker` or `export CONTAINER_ENGINE=podman`.\n\n<details>\n<summary><b>Install Docker or Podman</b> (macOS / Linux / Windows)</summary>\n\n#### macOS\n\n```bash\n\n---\n\n# Option A — Docker Desktop (easiest, includes GUI)\nbrew install --cask docker\nopen -a Docker                 # launch and wait for the whale icon to settle\n\n---\n\n# Option B — Docker\nsudo apt install -y docker.io\nsudo usermod -aG docker $USER  # log out / back in so your shell picks up the group\n```\n\n> **Rootful Docker ownership note:** with classic `sudo`-docker, files extracted from containers land owned by `root` on the host. ClawBench's driver detects this after each run and chowns `test-output/` back to your user automatically — but if you run other container tooling alongside, rootless Podman (or rootless Docker) avoids the issue entirely.\n\n#### Windows\n\n```powershell\n\n---\n\n# Option A — Docker Desktop (WSL2 backend)\nwinget install Docker.DockerDesktop\n\n---\n\n# then launch Docker Desktop from the Start menu and wait for it to be ready\n\n---\n\n## License & Acknowledgments\n\nApache 2.0 -- see [LICENSE](LICENSE).\n\nThe converted Claw-Eval suite in [`test-cases/claw-eval/`](test-cases/claw-eval/) is derived from [claw-eval/claw-eval](https://github.com/claw-eval/claw-eval) and the [claw-eval/Claw-Eval](https://huggingface.co/datasets/claw-eval/Claw-Eval) dataset, which are released under the MIT License. Third-party package notices are in [NOTICE](NOTICE).\n\nBuilt with [OpenClaw](https://github.com/openclaw/openclaw), [opencode](https://opencode.ai), [Claude Code](https://docs.anthropic.com/en/docs/claude-code), the [Claude in Chrome](https://code.claude.com/docs/en/chrome) extension, [OpenAI Codex CLI](https://github.com/openai/codex), [browser-use](https://github.com/browser-use/browser-use), [claw-code](https://github.com/ultraworkers/claw-code), [Hermes Agent](https://github.com/NousResearch/hermes-agent), [Pi](https://pi.dev/) with [pi-browser-harness](https://pi.dev/packages/pi-browser-harness), and [WebBrain](https://github.com/webbrain-one/webbrain) (selectable harnesses), [Microsoft Playwright MCP](https://github.com/microsoft/playwright-mcp) (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), [LiteLLM](https://github.com/BerriAI/litellm) (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), [noVNC](https://github.com/novnc/noVNC) (MPL 2.0), and [websockify](https://github.com/novnc/websockify) (LGPL 3.0).","github_created_at":"2026-04-10T01:59:17+00:00","created_at":"2026-07-11T12:00:01.505485+00:00","updated_at":"2026-09-20T06:58:06.803372+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agent-evaluation","name":"agent-evaluation"},{"slug":"agentic-ai","name":"agentic-ai"},{"slug":"browser-agent","name":"browser-agent"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"real-world-benchmark","name":"real-world-benchmark"},{"slug":"web-agents","name":"web-agents"}],"trust":{"provenance":{"is_fork":false,"github_id":1206519531,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T06:58:04.325Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":6,"days_since_push":0,"last_release_at":"2026-08-30T22:43:14Z","stars_delta_30d":264,"open_issues_delta_30d":-1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:00:02.721Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T06:58:05.349Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-20T06:58:05.349Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T06:58:05.349Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-20T06:58:05.349Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires setup for browser automation tasks within the Chrome environment."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["You are developing a browser AI agent and wish to measure its performance against everyday online activities, as ClawBench specifically simulates these scenarios.","Your project involves the use of Chrome extensions in conjunction with LLMS, requiring benchmarking that aligns with this specific integration."],"when_not_to_use":["If your focus is solely on backend server-based AI evaluations without a browser interface involvement, ClawBench will not be the appropriate choice.","For those developing standalone applications or mobile agents, ClawBench’s browser-centric tasks will not reflect their operational capabilities accurately."],"source":"enrich:decision_facts","observed_at":"2026-07-16T22:10:29.445Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Requires setup for browser automation tasks within the Chrome environment."},{"label":"Adopt for","value":"ClawBench offers an open-source benchmark framework for evaluating browser-based AI agents on real-world online tasks."},{"label":"License detail","value":"ClawBench operates under the Apache-2.0 license, offering a permissive free software license that encourages software reuse and interoperability."}]}}