GTA: General Tool Agent Benchmark and Evaluation Framework

[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents

[arXiv 2026] GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

🌟 Introduction

GTA-2 is a benchmark and evaluation kit for General Tool Agents, designed to bridge atomic tool-use evaluation and open-ended workflow evaluation in one repository.

Benchmark hierarchy

• GTA-Workflow: the new focus of GTA-2, for long-horizon, open-ended workflow evaluation.
• GTA-Atomic: the original GTA benchmark for short-horizon atomic tool-use tasks. Please refer to README_GTA-1.md.

This readme is centered around GTA-Workflow, which targets realistic long-horizon tasks with open-ended deliverables. Compared with traditional benchmark-style evaluation, GTA-Workflow focuses more on what an agent can finally accomplish in a complete workflow, rather than only whether it predicts the next tool call correctly.

What this repo supports

• Workflow-oriented agent evaluation.
  Evaluate long-horizon, open-ended agent tasks with deliverable-centric scoring.

• Both model and harness evaluation.
  GTA-Workflow is designed to evaluate not only the underlying LLM, but also the execution harness / agent framework behind it.

• Default OpenCompass-based evaluation.
  We provide a standard evaluation pipeline based on OpenCompass + Lagent, suitable for agents integrated as callable frameworks.

• Custom agent / custom LLM integration.
  Beyond the default setup, users can plug in their own agent framework or LLM backend. See docs/ADDING_NEW_AGENT_OR_LLM.md.

• End-to-end evaluation without OpenCompass.
  For agent products or closed systems that cannot be directly integrated into our framework, GTA-2 also supports evaluating final execution results directly, enabling assessment of systems such as Manus, Kortix, or OpenClaw.

📣 What's New

• [2026.4.20] Release GTA-2 paper and GTA-Workflow dataset. 🔥🔥🔥
• [2026.4.12] Release GTA-2, extending the original GTA benchmark into a hierarchical evaluation repo with:
  • GTA-Workflow for long-horizon, open-ended workflow evaluation in productivity scenarios,
  • support for evaluating both LLM capability (GPT, Gemini, Claude, etc.) and agent execution harnesses (OpenClaw, Manus, Kortix, etc.),
  • support for both OpenCompass-based agent evaluation and end-to-end result evaluation for external/closed agent systems.
• [2026.2.14] Update 🏆Leaderboard, Feb. 2026, including new models such as GPT-5, Gemini-2.5, Claude-4.5, Kimi-K2, Grok-4, Llama-4, Deepseek-V3.2, Qwen3-235B-A22B series.
• [2025.3.25] Update 🏆Leaderboard, Mar. 2025, including new models such as Deepseek-R1, Deepseek-V3, Qwen-QwQ, Qwen-2.5-max series.
• [2024.9.26] GTA is accepted to NeurIPS 2024 Dataset and Benchmark Track! 🎉🎉🎉
• [2024.7.11] Paper available on arXiv. ✨✨✨
• [2024.7.3] Release the evaluation and tool deployment code of GTA. 🔥🔥🔥
• [2024.7.1] Release the GTA dataset on Hugging Face. 🎉🎉🎉

📚 Dataset Statistics

GTA-Workflow: Real-World Productivity Tasks

GTA-Workflow focuses on long-horizon, open-ended productivity scenarios, where agents are required to complete realistic deliverabl