GraphCanon updated 3w · GitHub synced 3w · 33 views this month
Decision brief
GTA (General Tool Agent) is a benchmark and evaluation framework that assesses general tool agents in both atomic tasks and open-ended workflows.
Good fit when
- Use GTA when you need to evaluate the performance of an agent across long-horizon, open-ended productivity scenarios.
- Leverage GTA for evaluating both the LLM capability (e.g., GPT, Gemini) and the execution harness (e.g., OpenClaw, Manus).
Avoid when
- Avoid using GTA if you are only interested in evaluating short-term tasks without focusing on full workflow outcomes.
- Do not rely exclusively on GTA for assessing performance if the context of your use case differs significantly from productivity workflows that GTA aims to evaluate.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (99d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install GTASimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A benchmark for evaluating general tool agents with a focus on open-ended workflows and atomic tool-use tasks.
Capability facts
- Languages
- python
Source: github.language · Jul 29, 2026
Categories
Tags
README
GTA: General Tool Agent Benchmark and Evaluation Framework
[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents
[arXiv 2026] GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
⬇️ Download Dataset Here: [GTA-Atomic] [GTA-Workflow]
🌟 Introduction
GTA-2 is a benchmark and evaluation kit for General Tool Agents, designed to bridge atomic tool-use evaluation and open-ended workflow evaluation in one repository.
Benchmark hierarchy
- GTA-Workflow: the new focus of GTA-2, for long-horizon, open-ended workflow evaluation.
- GTA-Atomic: the original GTA benchmark for short-horizon atomic tool-use tasks. Please refer to README_GTA-1.md.
This readme is centered around GTA-Workflow, which targets realistic long-horizon tasks with open-ended deliverables. Compared with traditional benchmark-style evaluation, GTA-Workflow focuses more on what an agent can finally accomplish in a complete workflow, rather than only whether it predicts the next tool call correctly.
What this repo supports
-
Workflow-oriented agent evaluation.
Evaluate long-horizon, open-ended agent tasks with deliverable-centric scoring. -
Both model and harness evaluation.
GTA-Workflow is designed to evaluate not only the underlying LLM, but also the execution harness / agent framework behind it. -
Default OpenCompass-based evaluation.
We provide a standard evaluation pipeline based on OpenCompass + Lagent, suitable for agents integrated as callable frameworks. -
Custom agent / custom LLM integration.
Beyond the default setup, users can plug in their own agent framework or LLM backend. See docs/ADDING_NEW_AGENT_OR_LLM.md. -
End-to-end evaluation without OpenCompass.
For agent products or closed systems that cannot be directly integrated into our framework, GTA-2 also supports evaluating final execution results directly, enabling assessment of systems such as Manus, Kortix, or OpenClaw.
📣 What's New
- [2026.4.20] Release GTA-2 paper and GTA-Workflow dataset. 🔥🔥🔥
- [2026.4.12] Release GTA-2, extending the original GTA benchmark into a hierarchical evaluation repo with:
- GTA-Workflow for long-horizon, open-ended workflow evaluation in productivity scenarios,
- support for evaluating both LLM capability (GPT, Gemini, Claude, etc.) and agent execution harnesses (OpenClaw, Manus, Kortix, etc.),
- support for both OpenCompass-based agent evaluation and end-to-end result evaluation for external/closed agent systems.
- [2026.2.14] Update 🏆Leaderboard, Feb. 2026, including new models such as GPT-5, Gemini-2.5, Claude-4.5, Kimi-K2, Grok-4, Llama-4, Deepseek-V3.2, Qwen3-235B-A22B series.
- [2025.3.25] Update 🏆Leaderboard, Mar. 2025, including new models such as Deepseek-R1, Deepseek-V3, Qwen-QwQ, Qwen-2.5-max series.
- [2024.9.26] GTA is accepted to NeurIPS 2024 Dataset and Benchmark Track! 🎉🎉🎉
- [2024.7.11] Paper available on arXiv. ✨✨✨
- [2024.7.3] Release the evaluation and tool deployment code of GTA. 🔥🔥🔥
- [2024.7.1] Release the GTA dataset on Hugging Face. 🎉🎉🎉
📚 Dataset Statistics
GTA-Workflow: Real-World Productivity Tasks
GTA-Workflow focuses on long-horizon, open-ended productivity scenarios, where agents are required to complete realistic deliverabl
For agents
This page has a .md twin and JSON over the API.