GraphCanon
Home/LLM Frameworks/autoarena
autoarena logo

autoarena

Enrichment pending
kolenaIO/autoarena

Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation

GitHub Website

GraphCanon updated today · GitHub synced today

108
Stars
9
Forks
4
Open issues
6
Watchers
1y
Last push
TypeScript Apache-2.0Created Aug 28, 2024

Trust & integrity

Full report
Maintenance
Dormant (571d since push)
As of today · Source: github_public_v1
Provenance
Not a fork · Organization account
As of today · Source: github_public_v1
Security (OSV)
No lockfile
As of today · Source: none

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Overview

Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation

Capability facts

Languages
typescript, python

Source: github.language+pyproject.toml · Jul 11, 2026

Categories

LLM FrameworksEvaluation & Observability

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Jul 11, 2026)

python -m autoarena
Source link

Tags

evaluationllmaitestinghacktoberfestragtypescriptllm-evaluation

README

🔥 Getting Started

Install from PyPI:

pip install autoarena

Run as a module and visit localhost:8899 in your browser:

python -m autoarena

With the application running, getting started is simple:

  1. Create a project via the UI.
  2. Add responses from a model by selecting a CSV file with prompt and response columns.
  3. Configure an automated judge via the UI. Note that most judges require credentials, e.g. X_API_KEY in the environment where you're running AutoArena.
  4. Add responses from a second model to kick off an automated judging task using the judges you configured in the previous step to decide which of the models you've uploaded provided a better response to a given prompt.

That's it! After these steps you're fully set up for automated evaluation on AutoArena.