GraphCanon updated Oct 19, 2026 · GitHub synced Oct 19, 2026
Decision brief
Unverified catalog guidance: Open-LLM-Leaderboard evaluates large language models on open-style questions using a GPT-4-based evaluator and aggregates results in an accessible leaderboard format.
Good fit when
- Unverified catalog guidance: You need to evaluate your LLM's performance on open-ended, human-like question formats across multiple datasets without setting up the evaluation process yourself.
- Unverified catalog guidance: Your use case requires trust in GPT-4 as the underlying evaluator for objective grading of model capabilities.
Avoid when
- Unverified catalog guidance: You are seeking evaluations solely based on closed-response or multiple-choice questions where specific answers can be easily verified by non-LLM means.
- Unverified catalog guidance: Your project has constraints against using commercial LLMs like GPT-4 for evaluation due to cost, licensing issues, or the need for open-source alternatives.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (843d since push)
- As of Oct 19, 2026
- Provenance
- Not a fork · Organization account
- As of Oct 19, 2026
- Security (OSV)
- No lockfile
- As of Jul 15, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
Package installation is not established from the available documentation. Check the source repository for setup instructions.
git clone https://github.com/vila-lab/open-llm-leaderboardSource repositorySimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Open-LLM-Leaderboard evaluates the performance of large language models (LLMs) on open-ended questions through an automated process using an LLM-based evaluator.
Capability facts
- MCP server
- Unknown; MCP server not established by this scan
Source: repo_scan · Oct 19, 2026
- Languages
- python
Source: github.language · Oct 19, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (readme_declaration_v2, Oct 19, 2026)
To use it first run: `pip install lm-eval`.Source link
Tags
README
<p align="center" width="100%" <img src="assets/logo.jpg" alt="OSQ" style="width: 28%; min width: 150px; display: block; margin: auto;" </p Open LLM Leaderboard: Open Style Question Evaluation <a href="https://arxiv.org/abs/2406.07545" <img src="https://img.shields.io/badge/arXiv...
For agents
This page has a .md twin and JSON over the API.