{"data":{"slug":"open-compass-gta","name":"GTA","tagline":"GTA: General Tool Agent Benchmark and Evaluation Framework","github_url":"https://github.com/open-compass/GTA","owner":"open-compass","repo":"GTA","owner_avatar_url":"https://avatars.githubusercontent.com/u/143521324?v=4","primary_language":"Python","stars":148,"forks":10,"topics":["llm-agent","llm-evaluation"],"archived":false,"github_pushed_at":"2026-04-20T06:28:54+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/open-compass-gta","markdown_url":"https://www.graphcanon.com/tools/open-compass-gta.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/open-compass-gta","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=open-compass-gta","description":"[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2","homepage_url":null,"license":"Apache-2.0","open_issues":2,"watchers":6,"ai_summary":"A benchmark for evaluating general tool agents with a focus on open-ended workflows and atomic tool-use tasks.","readme_excerpt":"# GTA: General Tool Agent Benchmark and Evaluation Framework \n### [[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents](https://proceedings.neurips.cc/paper_files/paper/2024/file/8a75ee6d4b2eb0b777f549a32a5a5c28-Paper-Datasets_and_Benchmarks_Track.pdf)\n### [[arXiv 2026] GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows](https://arxiv.org/pdf/2604.15715)\n
\n\n⬇️ Download Dataset Here:\n[[GTA-Atomic](https://github.com/open-compass/GTA/releases/download/v0.1.0/gta_dataset.zip)]\n[[GTA-Workflow](https://github.com/open-compass/GTA/releases/download/v0.2.0/gta_workflow_dataset.zip)]\n
\n\n## 🌟 Introduction\n\nGTA-2 is a benchmark and evaluation kit for **General Tool Agents**, designed to bridge **atomic tool-use evaluation** and **open-ended workflow evaluation** in one repository.\n\n### Benchmark hierarchy\n\n- **GTA-Workflow**: the new focus of GTA-2, for long-horizon, open-ended workflow evaluation.\n- **GTA-Atomic**: the original GTA benchmark for short-horizon atomic tool-use tasks. Please refer to [README_GTA-1.md](README_GTA-1.md).\n\n
\n \n
\n\nThis readme is centered around **GTA-Workflow**, which targets realistic long-horizon tasks with open-ended deliverables. Compared with traditional benchmark-style evaluation, GTA-Workflow focuses more on **what an agent can finally accomplish in a complete workflow**, rather than only whether it predicts the next tool call correctly.\n\n### What this repo supports\n\n- **Workflow-oriented agent evaluation.** \n Evaluate long-horizon, open-ended agent tasks with deliverable-centric scoring.\n\n- **Both model and harness evaluation.** \n GTA-Workflow is designed to evaluate not only the underlying LLM, but also the **execution harness / agent framework** behind it.\n\n- **Default OpenCompass-based evaluation.** \n We provide a standard evaluation pipeline based on **[OpenCompass](https://github.com/open-compass/opencompass) + [Lagent](https://github.com/InternLM/lagent)**, suitable for agents integrated as callable frameworks.\n\n- **Custom agent / custom LLM integration.** \n Beyond the default setup, users can plug in their own agent framework or LLM backend. See [docs/ADDING_NEW_AGENT_OR_LLM.md](docs/ADDING_NEW_AGENT_OR_LLM.md).\n\n- **End-to-end evaluation without OpenCompass.** \n For agent products or closed systems that cannot be directly integrated into our framework, GTA-2 also supports evaluating **final execution results directly**, enabling assessment of systems such as **Manus, Kortix, or OpenClaw**.\n\n\n\n\n\n\n## πŸ“£ What's New\n- **[2026.4.20]** Release GTA-2 paper and GTA-Workflow dataset. πŸ”₯πŸ”₯πŸ”₯\n- **[2026.4.12]** Release **GTA-2**, extending the original GTA benchmark into a **hierarchical evaluation repo** with:\n - **GTA-Workflow** for long-horizon, open-ended workflow evaluation in productivity scenarios,\n - support for evaluating both **LLM capability (GPT, Gemini, Claude, etc.)** and **agent execution harnesses (OpenClaw, Manus, Kortix, etc.)**,\n - support for both **OpenCompass-based agent evaluation** and **end-to-end result evaluation** for external/closed agent systems.\n- **[2026.2.14]** Update πŸ†Leaderboard, Feb. 2026, including new models such as GPT-5, Gemini-2.5, Claude-4.5, Kimi-K2, Grok-4, Llama-4, Deepseek-V3.2, Qwen3-235B-A22B series.\n- **[2025.3.25]** Update πŸ†Leaderboard, Mar. 2025, including new models such as Deepseek-R1, Deepseek-V3, Qwen-QwQ, Qwen-2.5-max series.\n- **[2024.9.26]** GTA is accepted to NeurIPS 2024 Dataset and Benchmark Track! πŸŽ‰πŸŽ‰πŸŽ‰\n- **[2024.7.11]** Paper available on arXiv. ✨✨✨\n- **[2024.7.3]** Release the evaluation and tool deployment code of GTA. πŸ”₯πŸ”₯πŸ”₯\n- **[2024.7.1]** Release the GTA dataset on Hugging Face. πŸŽ‰πŸŽ‰πŸŽ‰\n\n## πŸ“š Dataset Statistics\n\n### GTA-Workflow: Real-World Productivity Tasks\nGTA-Workflow focuses on **long-horizon, open-ended productivity scenarios**, where agents are required to complete realistic deliverabl","github_created_at":"2024-06-06T11:46:09+00:00","created_at":"2026-07-11T12:02:08.950125+00:00","updated_at":"2026-07-29T00:00:22.620451+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"evaluation-framework","name":"evaluation-framework"},{"slug":"general-tool-agents","name":"general tool agents"},{"slug":"workflow-evaluation","name":"workflow evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":811330900,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-29T00:00:21.740Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":99,"last_release_at":"2026-04-20T06:27:27Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:02:16.816Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-29T00:00:22.324Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-29T00:00:22.324Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-29T00:00:22.324Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use GTA when you need to evaluate the performance of an agent across long-horizon, open-ended productivity scenarios.","Leverage GTA for evaluating both the LLM capability (e.g., GPT, Gemini) and the execution harness (e.g., OpenClaw, Manus).","Consider GTA if your project involves integrating or testing a specific LLM backend with custom agent frameworks."],"when_not_to_use":["Avoid using GTA if you are only interested in evaluating short-term tasks without focusing on full workflow outcomes.","Do not rely exclusively on GTA for assessing performance if the context of your use case differs significantly from productivity workflows that GTA aims to evaluate.","highlevel_tags"],"source":"enrich:decision_facts","observed_at":"2026-07-12T14:26:44.558Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"GTA (General Tool Agent) is a benchmark and evaluation framework that assesses general tool agents in both atomic tasks and open-ended workflows."}]}}