every_eval_ever logo

every_eval_ever

evaleval/every_eval_ever

Shared schema and crowdsourced eval database

GraphCanon updated Oct 18, 2026 · GitHub synced Oct 18, 2026

137 stars53 forksLast push Oct 12, 2026 Python MIT

Decision brief

Unverified catalog guidance: Every Eval Ever is dedicated to providing a standardized metadata framework and a crowdsourced evaluation database for AI results.

Good fit when

  • Unverified catalog guidance: Use Every Eval Ever if you need to compare evaluation results from different frameworks in a consistent manner, ensuring results can be easily reproduced or reused as they conform to a defined schema.
  • Unverified catalog guidance: Opt for this project when your workflow involves leaderboard scrapes and research paper data that you want to integrate with local runs of AI models.

Avoid when

  • Unverified catalog guidance: Avoid Every Eval Ever if you require real-time updates on evaluation results, as the database relies on contributions from a community to maintain and update its dataset.
  • Unverified catalog guidance: If your project needs to integrate evaluation outcomes without an explicit need for extensive metadata validation or standardization, this tool might be less suitable.
Pricing:
freemium - Every Eval Ever is open-source under the MIT license, allowing free use and modification. No direct costs are associated with using the schema or contributing to the database.
Requirements:
Min 2 GB RAM; To utilize all features, you need to install specific converter dependencies via pip.

Observed Jul 16, 2026 · Source: enrich:decision_facts

Verify the decision

Maintenance and security

Full trust report
Maintenance
Very active (5d since push)
As of Oct 18, 2026
Provenance
Not a fork · Organization account
As of Oct 18, 2026
Security (OSV)
No lockfile
As of Jul 15, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

Package installation is not established from the available documentation. Check the source repository for setup instructions.

git clone https://github.com/evaleval/every_eval_ever
Source repository

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Every Eval Ever defines a standardized metadata format for storing AI evaluation results from various sources including leaderboard scrapes, research papers, and local runs.

Capability facts

CLI
Declared CLI entrypoint

Source: pyproject.toml:[project.scripts] · Oct 18, 2026

MCP server
Unknown; MCP server not established by this scan

Source: repo_scan · Oct 18, 2026

Languages
python

Source: github.language+pyproject.toml+.github/ci-types/pyproject.toml · Oct 18, 2026

Categories

Tags

README

Every Eval Ever EvalEval Coalition — "We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations." 📄 Paper (arXiv:2606.14516) Every Eval Ever is a shared schema and crowdsourced eval datab...

For agents

This page has a .md twin and JSON over the API.

Was this helpful?

Anonymous feedback helps us improve pages and translations.