{"data":{"slug":"open-compass-gta","name":"GTA","tagline":"[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2","github_url":"https://github.com/open-compass/GTA","owner":"open-compass","repo":"GTA","owner_avatar_url":"https://avatars.githubusercontent.com/u/143521324?v=4","primary_language":"Python","stars":147,"forks":10,"topics":["llm-agent","llm-evaluation"],"archived":false,"github_pushed_at":"2026-04-20T06:28:54+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/open-compass-gta","markdown_url":"https://www.graphcanon.com/tools/open-compass-gta.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/open-compass-gta","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=open-compass-gta","description":"[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2","homepage_url":null,"license":"Apache-2.0","open_issues":1,"watchers":6,"ai_summary":null,"readme_excerpt":"# GTA: General Tool Agent Benchmark and Evaluation Framework \n### [[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents](https://proceedings.neurips.cc/paper_files/paper/2024/file/8a75ee6d4b2eb0b777f549a32a5a5c28-Paper-Datasets_and_Benchmarks_Track.pdf)\n### [[arXiv 2026] GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows](https://arxiv.org/pdf/2604.15715)\n
\n\n⬇️ Download Dataset Here:\n[[GTA-Atomic](https://github.com/open-compass/GTA/releases/download/v0.1.0/gta_dataset.zip)]\n[[GTA-Workflow](https://github.com/open-compass/GTA/releases/download/v0.2.0/gta_workflow_dataset.zip)]\n
\n\n## 🌟 Introduction\n\nGTA-2 is a benchmark and evaluation kit for **General Tool Agents**, designed to bridge **atomic tool-use evaluation** and **open-ended workflow evaluation** in one repository.\n\n### Benchmark hierarchy\n\n- **GTA-Workflow**: the new focus of GTA-2, for long-horizon, open-ended workflow evaluation.\n- **GTA-Atomic**: the original GTA benchmark for short-horizon atomic tool-use tasks. Please refer to [README_GTA-1.md](README_GTA-1.md).\n\n
\n \n
\n\nThis readme is centered around **GTA-Workflow**, which targets realistic long-horizon tasks with open-ended deliverables. Compared with traditional benchmark-style evaluation, GTA-Workflow focuses more on **what an agent can finally accomplish in a complete workflow**, rather than only whether it predicts the next tool call correctly.\n\n### What this repo supports\n\n- **Workflow-oriented agent evaluation.** \n Evaluate long-horizon, open-ended agent tasks with deliverable-centric scoring.\n\n- **Both model and harness evaluation.** \n GTA-Workflow is designed to evaluate not only the underlying LLM, but also the **execution harness / agent framework** behind it.\n\n- **Default OpenCompass-based evaluation.** \n We provide a standard evaluation pipeline based on **[OpenCompass](https://github.com/open-compass/opencompass) + [Lagent](https://github.com/InternLM/lagent)**, suitable for agents integrated as callable frameworks.\n\n- **Custom agent / custom LLM integration.** \n Beyond the default setup, users can plug in their own agent framework or LLM backend. See [docs/ADDING_NEW_AGENT_OR_LLM.md](docs/ADDING_NEW_AGENT_OR_LLM.md).\n\n- **End-to-end evaluation without OpenCompass.** \n For agent products or closed systems that cannot be directly integrated into our framework, GTA-2 also supports evaluating **final execution results directly**, enabling assessment of systems such as **Manus, Kortix, or OpenClaw**.\n\n\n\n\n\n\n## πŸ“£ What's New\n- **[2026.4.20]** Release GTA-2 paper and GTA-Workflow dataset. πŸ”₯πŸ”₯πŸ”₯\n- **[2026.4.12]** Release **GTA-2**, extending the original GTA benchmark into a **hierarchical evaluation repo** with:\n - **GTA-Workflow** for long-horizon, open-ended workflow evaluation in productivity scenarios,\n - support for evaluating both **LLM capability (GPT, Gemini, Claude, etc.)** and **agent execution harnesses (OpenClaw, Manus, Kortix, etc.)**,\n - support for both **OpenCompass-based agent evaluation** and **end-to-end result evaluation** for external/closed agent systems.\n- **[2026.2.14]** Update πŸ†Leaderboard, Feb. 2026, including new models such as GPT-5, Gemini-2.5, Claude-4.5, Kimi-K2, Grok-4, Llama-4, Deepseek-V3.2, Qwen3-235B-A22B series.\n- **[2025.3.25]** Update πŸ†Leaderboard, Mar. 2025, including new models such as Deepseek-R1, Deepseek-V3, Qwen-QwQ, Qwen-2.5-max series.\n- **[2024.9.26]** GTA is accepted to NeurIPS 2024 Dataset and Benchmark Track! πŸŽ‰πŸŽ‰πŸŽ‰\n- **[2024.7.11]** Paper available on arXiv. ✨✨✨\n- **[2024.7.3]** Release the evaluation and tool deployment code of GTA. πŸ”₯πŸ”₯πŸ”₯\n- **[2024.7.1]** Release the GTA dataset on Hugging Face. πŸŽ‰πŸŽ‰πŸŽ‰\n\n## πŸ“š Dataset Statistics\n\n### GTA-Workflow: Real-World Productivity Tasks\nGTA-Workflow focuses on **long-horizon, open-ended productivity scenarios**, where agents are required to complete realistic deliverabl","github_created_at":"2024-06-06T11:46:09+00:00","created_at":"2026-07-11T12:02:08.950125+00:00","updated_at":"2026-07-11T12:02:20.602075+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"python","name":"python"},{"slug":"llm-agent","name":"llm-agent"},{"slug":"llm-evaluation","name":"llm evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":811330900,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T12:02:09.711Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":1,"days_since_push":82,"last_release_at":"2026-04-20T06:27:27Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:02:16.816Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T12:02:16.397Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T12:02:16.397Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T12:02:16.397Z"}}}}