{"data":{"slug":"kolenaio-autoarena","name":"autoarena","tagline":"Automated evaluation of LLMs and RAG systems","github_url":"https://github.com/kolenaIO/autoarena","owner":"kolenaIO","repo":"autoarena","owner_avatar_url":"https://avatars.githubusercontent.com/u/77010818?v=4","primary_language":"TypeScript","stars":108,"forks":9,"topics":["ai","evaluation","hacktoberfest","llm","llm-evaluation","rag","testing"],"archived":false,"github_pushed_at":"2024-12-16T12:25:44+00:00","maintenance_label":"Dormant","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/kolenaio-autoarena","markdown_url":"https://www.graphcanon.com/tools/kolenaio-autoarena.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/kolenaio-autoarena","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=kolenaio-autoarena","description":"Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation","homepage_url":"https://www.kolena.com/autoarena/","license":"Apache-2.0","open_issues":4,"watchers":6,"ai_summary":"autoarena is a tool to rank LLMs and Retrieval-Augmented Generation (RAG) systems using automated head-to-head evaluations.","readme_excerpt":"## 🔥 Getting Started\n\nInstall from [PyPI](https://pypi.org/project/autoarena/):\n\n```shell\npip install autoarena\n```\n\nRun as a module and visit [localhost:8899](http://localhost:8899/) in your browser:\n\n```shell\npython -m autoarena\n```\n\nWith the application running, getting started is simple:\n\n1. Create a project via the UI.\n1. Add responses from a model by selecting a CSV file with `prompt` and `response` columns.\n1. Configure an automated judge via the UI. Note that most judges require credentials, e.g. `X_API_KEY` in the\n   environment where you're running AutoArena.\n1. Add responses from a second model to kick off an automated judging task using the judges you configured in the\n   previous step to decide which of the models you've uploaded provided a better `response` to a given `prompt`.\n\nThat's it! After these steps you're fully set up for automated evaluation on AutoArena.","github_created_at":"2024-08-28T19:55:28+00:00","created_at":"2026-07-11T12:03:09.066972+00:00","updated_at":"2026-09-20T07:01:54.425126+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"evaluation","name":"evaluation"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"rag","name":"rag"},{"slug":"testing","name":"testing"}],"trust":{"provenance":{"is_fork":false,"github_id":849004809,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T07:01:52.494Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":642,"last_release_at":"2024-10-07T13:52:21Z","stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:03:16.691Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T07:01:53.553Z"},"languages":{"value":["typescript","python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T07:01:53.553Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-20T07:01:53.553Z"}},"decision_facts":{"hosting":{"model":"self_hosted"},"pricing":null,"requirements":{"notes":["Python environment and internet access are needed for PyPI installation via pip."],"requires_docker":false},"constraints":{"hosting_model":"self_hosted","requires_docker":false},"when_to_use":["When you need a TypeScript-based tool to rank LLMs and RAG systems via automated head-to-head comparisons, and a web UI is preferable.","If your project relies on PyPI installation and requires a simple setup using environment variables for API credentials."],"when_not_to_use":["If your environment lacks the necessary Python packages or you cannot install from PyPI due to restrictions.","When real-time evaluation needs surpass capabilities, such as requiring immediate feedback beyond autoarena's batch-processing approach."],"source":"enrich:decision_facts","observed_at":"2026-07-16T23:08:39.875Z"},"constraint_facets":{"hosting_model":"self_hosted","requires_docker":false},"decision_summary":[{"label":"Hosting","value":"self hosted"},{"label":"Requirements","value":"Python environment and internet access are needed for PyPI installation via pip."},{"label":"Adopt for","value":"autoarena automates evaluations for LLMs and RAG systems through a user-friendly interface where projects are created and judged without manual intervention by the users."},{"label":"License detail","value":"Apache-2.0 license"}]}}