GAGE is a unified, extensible evaluation framework for large language models, multimodal models, audio models, diffusion models, agents, and game environments. It provides one evaluation engine for datasets, model backends, metrics, arena runtimes, structured outputs, and replayable artifacts.

Game Arena Showcase

Why GAGE?

• Fast evaluation engine: Run local smoke tests, model-backed jobs, and larger benchmark batches through the same pipeline shape.
• Unified evaluation surface: Datasets, backends, role adapters, metrics, and output contracts are configured instead of hand-wired per benchmark.
• Game and agent sandboxing: Game Arena, AgentKitV2, AppWorld, SWE-bench-style agent tasks, GUI interaction, and tool-augmented workflows share the same run/output model.
• External harness integration: Delegate task-batch benchmarks to Harbor, then import trial evidence back into standard GAGE samples, metrics, reports, and raw artifacts.
• Replayable GameKit runtime: Gomoku, Tic-Tac-Toe, Doudizhu, Mahjong, PettingZoo Space Invaders, Retro Mario, and ViZDoom now emit structured arena traces plus arena_visual sessions.
• Operational visibility: Runs write summary.json, sample outputs, logs, visual artifacts, and a static report pack so failures can be inspected after the fact.

Design Overview

Core design philosophy: everything is a step, everything is configurable.

Architecture Design

Orchestration Design

Game Arena Design

AgentKitV2 Design

External Harness Design

Quick Start

1. Installation

# If you are in the mono-repo root:
cd gage-eval-main

# Python 3.10+ recommended
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For Game Arena LLM configs, use the *_openai_gamekit.yaml variants and export OPENAI_API_KEY. The model defaults to gpt-5.4; set GAGE_GAME_ARENA_LLM_MODEL to override it, or set OPENAI_API_BASE for an OpenAI-compatible endpoint.

2. Run a Basic Demo

python run.py \
  --config config/run_configs/demo_echo_run_1.yaml \
  --output-dir runs \
  --run-id demo_echo

3. View Reports

Default output structure:

runs/<run_id>/
  events.jsonl
  samples.jsonl
  summary.json
  samples/
    <namespace>/
      <sample_id>.json
  report_pack/
    report.html
    report_context.json
    report_context.md
    prompt.txt
    diagnostics.json
    assets_manifest.json

Open runs/<run_id>/report_pack/report.html for the execution-aware report: primary metrics, key findings, scenario profiles, evidence links, media previews, diagnostics, an