Home/Evaluation & Observability/Open-LLM-Leaderboard
Open-LLM-Leaderboard logo

Open-LLM-Leaderboard

VILA-Lab/Open-LLM-Leaderboard

Tracks LLM performance on open-style questions

GraphCanon updated Oct 19, 2026 · GitHub synced Oct 19, 2026

53 stars7 forksLast push Jun 27, 2024 Python CC-BY-4.0

Decision brief

Unverified catalog guidance: Open-LLM-Leaderboard evaluates large language models on open-style questions using a GPT-4-based evaluator and aggregates results in an accessible leaderboard format.

Good fit when

  • Unverified catalog guidance: You need to evaluate your LLM's performance on open-ended, human-like question formats across multiple datasets without setting up the evaluation process yourself.
  • Unverified catalog guidance: Your use case requires trust in GPT-4 as the underlying evaluator for objective grading of model capabilities.

Avoid when

  • Unverified catalog guidance: You are seeking evaluations solely based on closed-response or multiple-choice questions where specific answers can be easily verified by non-LLM means.
  • Unverified catalog guidance: Your project has constraints against using commercial LLMs like GPT-4 for evaluation due to cost, licensing issues, or the need for open-source alternatives.

Observed Jul 16, 2026 · Source: enrich:decision_facts

Verify the decision

Maintenance and security

Full trust report
Maintenance
Dormant (843d since push)
As of Oct 19, 2026
Provenance
Not a fork · Organization account
As of Oct 19, 2026
Security (OSV)
No lockfile
As of Jul 15, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

Package installation is not established from the available documentation. Check the source repository for setup instructions.

git clone https://github.com/vila-lab/open-llm-leaderboard
Source repository

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Open-LLM-Leaderboard evaluates the performance of large language models (LLMs) on open-ended questions through an automated process using an LLM-based evaluator.

Capability facts

MCP server
Unknown; MCP server not established by this scan

Source: repo_scan · Oct 19, 2026

Languages
python

Source: github.language · Oct 19, 2026

Categories

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Documented Python runtime (not runtime-tested)Python

Source: README excerpt (readme_declaration_v2, Oct 19, 2026)

To use it first run: `pip install lm-eval`.
Source link

Tags

README

<p align="center" width="100%" <img src="assets/logo.jpg" alt="OSQ" style="width: 28%; min width: 150px; display: block; margin: auto;" </p Open LLM Leaderboard: Open Style Question Evaluation <a href="https://arxiv.org/abs/2406.07545" <img src="https://img.shields.io/badge/arXiv...

For agents

This page has a .md twin and JSON over the API.

Was this helpful?

Anonymous feedback helps us improve pages and translations.