Home/Compare/auto-evaluator vs Open-LLM-Leaderboard

Comparison

auto-evaluator vs Open-LLM-Leaderboard

Verdict

Unverified catalog guidance: auto-evaluator: Auto-evaluator is a Python-based tool designed for evaluating LLM QA chains with the capability to auto-generate question-answer pairs from user-provided documents and evaluate answers using configurations chosen via UI. Open-LLM-Leaderboard: Open-LLM-Leaderboard evaluates large language models on open-style questions using a GPT-4-based evaluator and aggregates results in an accessible leaderboard format.

Markdown twin · auto-evaluator alternatives · Open-LLM-Leaderboard alternatives

GraphCanon updated Oct 19, 2026

auto-evaluator logo

auto-evaluator

rlancemartin/auto-evaluator

★ 1.1kpushed May 10, 2023
vs
Open-LLM-Leaderboard logo

Open-LLM-Leaderboard

VILA-Lab/Open-LLM-Leaderboard

★ 53pushed Jun 27, 2024

Trust & integrity

Signalauto-evaluatorOpen-LLM-Leaderboard
Maintenance
Dormant (1255d since push)
As of Oct 16, 2026 · github_public_v1
Dormant (843d since push)
As of Oct 19, 2026 · github_public_v1
Provenance
Not a fork · Personal account
As of Oct 16, 2026 · github_public_v1
Not a fork · Organization account
As of Oct 19, 2026 · github_public_v1
OSV dependency advisories
Published findings
As of Jul 11, 2026 · osv@v1
No lockfile (source not queried)
As of Jul 15, 2026 · osv@v1
deps.dev advisories
Not queried
deps.dev@v1
Not queried
deps.dev@v1
OpenSSF Scorecard
Not queried
openssf-scorecard@v1
Not queried
openssf-scorecard@v1

Tagline

auto-evaluator
A lightweight evaluation tool for question-answering using Langchain
Open-LLM-Leaderboard
Tracks LLM performance on open-style questions

Stars

auto-evaluator
1.1k
Open-LLM-Leaderboard
53

Forks

auto-evaluator
92
Open-LLM-Leaderboard
7

Open issues

auto-evaluator
3
Open-LLM-Leaderboard
1

Language

auto-evaluator
Python
Open-LLM-Leaderboard
Python

Adopt for

auto-evaluator
Unverified catalog guidance: Auto-evaluator is a Python-based tool designed for evaluating LLM QA chains with the capability to auto-generate question-answer pairs from user-provided documents and evaluate answers using configurations chosen via UI.
Open-LLM-Leaderboard
Unverified catalog guidance: Open-LLM-Leaderboard evaluates large language models on open-style questions using a GPT-4-based evaluator and aggregates results in an accessible leaderboard format.

Persona

auto-evaluator
-
Open-LLM-Leaderboard
-

Runtime

auto-evaluator
-
Open-LLM-Leaderboard
-

License

auto-evaluator
-
Open-LLM-Leaderboard
CC-BY-4.0

Last pushed

auto-evaluator
May 10, 2023
Open-LLM-Leaderboard
Jun 27, 2024

Categories

auto-evaluator
Evaluation & Observability
Open-LLM-Leaderboard
Evaluation & Observability

Trust and health

Days since push

auto-evaluator
1255d
Open-LLM-Leaderboard
843d

Open issues (recorded)

auto-evaluator
3
Open-LLM-Leaderboard
1

Stars delta

auto-evaluator
+1 (30d)
Open-LLM-Leaderboard
0 (30d)

Owner type

auto-evaluator
User
Open-LLM-Leaderboard
Organization

OSV dependency advisories

auto-evaluator
Published findings
Open-LLM-Leaderboard
No lockfile (source not queried)

Full report

auto-evaluator
Trust report
Open-LLM-Leaderboard
Trust report

Shared compatibility

  • Python · auto-evaluator: Documented Python runtime (not runtime-tested) · Open-LLM-Leaderboard: Documented Python runtime (not runtime-tested)

Choose auto-evaluator if…

  • Tags unique to auto-evaluator: evaluation, gpt-3.5-turbo, langchain, llm.
  • Unverified catalog guidance: Use when you need a lightweight solution for testing question-answering capabilities of Langchain models.
  • More GitHub stars (1.1k vs 53) - visibility, not fit.

When NOT to use auto-evaluator

  • Unverified catalog guidance: Avoid using this tool when you do not have access to an OpenAI API key providing access to GPT-4, as it uses that by default for optimal settings.
  • Unverified catalog guidance: If you are looking for a tool that does not require you to input documents for question generation and prefer a more customized prompt approach rather than the auto-generation feature.

Choose Open-LLM-Leaderboard if…

  • Tags unique to Open-LLM-Leaderboard: leaderboard, llm-evaluation, model-performance-tracking, open-style-questions.
  • Unverified catalog guidance: You need to evaluate your LLM's performance on open-ended, human-like question formats across multiple datasets without setting up the evaluation process yourself.
  • More recently updated (last pushed Jun 27, 2024).

When NOT to use Open-LLM-Leaderboard

  • Unverified catalog guidance: You are seeking evaluations solely based on closed-response or multiple-choice questions where specific answers can be easily verified by non-LLM means.
  • Unverified catalog guidance: Your project has constraints against using commercial LLMs like GPT-4 for evaluation due to cost, licensing issues, or the need for open-source alternatives.

Explore

Sources

Every stat on this page traces to a dated GitHub sync, license file, enrichment field, or trust scan.

GitHub stars on cards: auto-evaluator 1.1k · Open-LLM-Leaderboard 53 (synced Oct 16, 2026).

Common questions

What is the difference between auto-evaluator and Open-LLM-Leaderboard?
auto-evaluator: A lightweight evaluation tool for question-answering using Langchain. Open-LLM-Leaderboard: Tracks LLM performance on open-style questions. See the comparison table for recorded GitHub metadata and shared categories.
When should I choose auto-evaluator over Open-LLM-Leaderboard?
For auto-evaluator, compare these recorded points with Open-LLM-Leaderboard: Tags unique to auto-evaluator: evaluation, gpt-3.5-turbo, langchain, llm; Unverified catalog guidance: Use when you need a lightweight solution for testing question-answering capabilities of Langchain models; More GitHub stars (1.1k vs 53) - visibility, not fit.
When should I choose Open-LLM-Leaderboard over auto-evaluator?
For Open-LLM-Leaderboard, compare these recorded points with auto-evaluator: Tags unique to Open-LLM-Leaderboard: leaderboard, llm-evaluation, model-performance-tracking, open-style-questions; Unverified catalog guidance: You need to evaluate your LLM's performance on open-ended, human-like question formats across multiple datasets without setting up the evaluation process yourself; More recently updated (last pushed Jun 27, 2024).
When should I avoid auto-evaluator?
Unverified catalog guidance: Avoid using this tool when you do not have access to an OpenAI API key providing access to GPT-4, as it uses that by default for optimal settings. Unverified catalog guidance: If you are looking for a tool that does not require you to input documents for question generation and prefer a more customized prompt approach rather than the auto-generation feature.
When should I avoid Open-LLM-Leaderboard?
Unverified catalog guidance: You are seeking evaluations solely based on closed-response or multiple-choice questions where specific answers can be easily verified by non-LLM means. Unverified catalog guidance: Your project has constraints against using commercial LLMs like GPT-4 for evaluation due to cost, licensing issues, or the need for open-source alternatives.
Is auto-evaluator or Open-LLM-Leaderboard more popular on GitHub?
auto-evaluator has more recorded GitHub stars (1,103 vs 53). Stars measure visibility, not whether either tool fits your constraints.
Which licenses are recorded for auto-evaluator and Open-LLM-Leaderboard?
auto-evaluator: license not recorded. Open-LLM-Leaderboard: CC-BY-4.0. Check each repository's current license and terms before use. Public GitHub hosting alone does not establish open-source licensing.
Where can I find alternatives to auto-evaluator or Open-LLM-Leaderboard?
GraphCanon lists candidates at auto-evaluator alternatives and Open-LLM-Leaderboard alternatives (auto-evaluator markdown twin, Open-LLM-Leaderboard markdown twin). The discovery order uses stored relationship proposals and shared categories; it is not a verified replacement ranking.
Is there a machine-readable version of this comparison?
Yes. The markdown twin at this comparison mirrors this page for agents and LLM crawlers, with the same stats table and FAQ answers.
Which is better maintained, auto-evaluator or Open-LLM-Leaderboard?
auto-evaluator: Dormant. Open-LLM-Leaderboard: Dormant. Compare maintenance labels, days since push, and release cadence in the trust section below - stars alone do not measure maintenance.
Where are the full trust reports for auto-evaluator and Open-LLM-Leaderboard?
GraphCanon publishes per-repo trust reports with dated maintenance, provenance, and scan summaries: auto-evaluator trust report; Open-LLM-Leaderboard trust report.

Was this helpful?

Anonymous feedback helps us improve pages and translations.