{"data":{"slug":"hithink-research-gage","name":"GAGE","tagline":"General AI evaluation and Gauge Engine. A unified evaluation engine for LLMs, MLLMs, audio, and diffusion models.","github_url":"https://github.com/HiThink-Research/GAGE","owner":"HiThink-Research","repo":"GAGE","owner_avatar_url":"https://avatars.githubusercontent.com/u/186997494?v=4","primary_language":"Python","stars":51,"forks":8,"topics":["agent","game-arena","llm","llm-evaluation","mllm-evaluation","sandbox-environment"],"archived":false,"github_pushed_at":"2026-06-02T06:49:47+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/hithink-research-gage","markdown_url":"https://www.graphcanon.com/tools/hithink-research-gage.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/hithink-research-gage","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=hithink-research-gage","description":"General AI evaluation and Gauge Engine. A unified evaluation engine for LLMs, MLLMs, audio, and diffusion models.","homepage_url":null,"license":null,"open_issues":3,"watchers":2,"ai_summary":null,"readme_excerpt":"
\n\n# 馃搻 GAGE: General AI evaluation and Gauge Engine\n\n\n\n\n\n\n**English** 路 [涓枃](README_zh.md)\n\n馃摟 **Contact:** [zhangrongjunchen@myhexin.com](mailto:zhangrongjunchen@myhexin.com)\n\n[Overview](docs/guide/framework_overview.md) 路 [Sample Schema](docs/guide/sample.md) 路 [Smart Defaults](docs/guide/smart_defaults.md) 路 [Run Reports](docs/guide/run_report_perception.md) 路 [Game Arena](docs/guide/game_arena.md) 路 [Arena Visual Control](docs/guide/game_arena_topics/game_arena_visual_control.md) 路 [AgentKitV2](docs/guide/agent_evaluation.md) 路 [External Harness](docs/guide/external_harness.md) 路 [Benchmark](docs/guide/benchmark.md) 路 [Contributing](CONTRIBUTING.md) 路 [Standards](AGENTS.md)\n\n
\n\n---\n\n**GAGE** is a unified, extensible evaluation framework for large language models, multimodal models, audio models, diffusion models, agents, and game environments. It provides one evaluation engine for datasets, model backends, metrics, arena runtimes, structured outputs, and replayable artifacts.\n\n## Game Arena Showcase\n\n

\"Gomoku\"Doudizhu\"Mahjong

\n\n

\n \"Space\n \"Mario\n \"VizDoom\n

\n\n## Why GAGE?\n\n- **Fast evaluation engine**: Run local smoke tests, model-backed jobs, and larger benchmark batches through the same pipeline shape.\n- **Unified evaluation surface**: Datasets, backends, role adapters, metrics, and output contracts are configured instead of hand-wired per benchmark.\n- **Game and agent sandboxing**: Game Arena, AgentKitV2, AppWorld, SWE-bench-style agent tasks, GUI interaction, and tool-augmented workflows share the same run/output model.\n- **External harness integration**: Delegate task-batch benchmarks to Harbor, then import trial evidence back into standard GAGE samples, metrics, reports, and raw artifacts.\n- **Replayable GameKit runtime**: Gomoku, Tic-Tac-Toe, Doudizhu, Mahjong, PettingZoo Space Invaders, Retro Mario, and ViZDoom now emit structured arena traces plus `arena_visual` sessions.\n- **Operational visibility**: Runs write `summary.json`, sample outputs, logs, visual artifacts, and a static [report pack](docs/guide/run_report_perception.md) so failures can be inspected after the fact.\n\n## Design Overview\n\n> Core design philosophy: everything is a step, everything is configurable.\n\n### Architecture Design\n\n\n\n### Orchestration Design\n\n\n\n### Game Arena Design\n\n\n\n### AgentKitV2 Design\n\n\n\n### External Harness Design\n\n\n\n## Quick Start\n\n### 1. Installation\n\n```bash\n# If you are in the mono-repo root:\ncd gage-eval-main\n\n# Python 3.10+ recommended\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\nFor Game Arena LLM configs, use the `*_openai_gamekit.yaml` variants and export `OPENAI_API_KEY`. The model defaults to `gpt-5.4`; set `GAGE_GAME_ARENA_LLM_MODEL` to override it, or set `OPENAI_API_BASE` for an OpenAI-compatible endpoint.\n\n### 2. Run a Basic Demo\n\n```bash\npython run.py \\\n --config config/run_configs/demo_echo_run_1.yaml \\\n --output-dir runs \\\n --run-id demo_echo\n```\n\n### 3. View Reports\n\nDefault output structure:\n\n```text\nruns//\n events.jsonl\n samples.jsonl\n summary.json\n samples/\n /\n .json\n report_pack/\n report.html\n report_context.json\n report_context.md\n prompt.txt\n diagnostics.json\n assets_manifest.json\n```\n\nOpen `runs//report_pack/report.html` for the execution-aware report: primary metrics, key findings, scenario profiles, evidence links, media previews, diagnostics, an","github_created_at":"2025-05-20T10:42:43+00:00","created_at":"2026-07-15T10:40:47.048774+00:00","updated_at":"2026-07-15T10:40:50.387258+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"agent","name":"agent"},{"slug":"game-arena","name":"game-arena"},{"slug":"llm","name":"llm"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"mllm-evaluation","name":"mllm-evaluation"},{"slug":"python","name":"python"},{"slug":"sandbox-environment","name":"sandbox-environment"}],"trust":{"provenance":{"is_fork":false,"github_id":986938784,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-15T10:40:47.975Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":3,"days_since_push":43,"last_release_at":"2026-06-02T06:33:53Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":89,"high_count":0,"last_scan_at":"2026-07-15T10:40:48.447Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-15T10:40:47.741Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-15T10:40:47.741Z"}}}}