Every Eval Ever

EvalEval Coalition — "We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations."

📖 Documentation

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused. The three components that make it work:

• 📋 A metadata schema (eval.schema.json) that defines the information needed for meaningful comparison of evaluation results, including instance-level data
• 🔧 Validation that checks data against the schema before it enters the repository
• 🔌 Converters for Inspect AI, HELM, and lm-eval-harness, so you can transform your existing evaluation logs into the standard format

Install the package:

pip install every-eval-ever

Optional converter dependencies:

pip install 'every-eval-ever[inspect]'
pip install 'every-eval-ever[helm]'
pip install 'every-eval-ever[all]'

Terminology

🚀 Contributor Guide

New data can be contributed to the Hugging Face Dataset using the following process:

Leaderboard/evaluation data is split-up into files by individual model, and data for each model is stored using eval.schema.json. The repository is structured into folders as data/{benchmark_name}/{developer_name}/{model_name}/.

TL;DR How to successfully submit

1. Data must conform to eval.schema.json (current version: 0.2.2)
2. The validation pipeline will automatically verify the data submitted in the pull request, but can also be manually triggered by typing /eee validate changed in a comment on the HF PR.
3. An EvalEval member will review and merge your submission

PR Naming Convention

Use these prefixes in your pull request titles:

• [Submission] - New evaluation data
• [Issue #N] - Fix for a specific GitHub issue
• [Feature] - New functionality not tied to an issue
• [Docs] - Documentation changes
• [ACL Shared Task] - Shared task submissions (priority review)

UUID Naming Convention

Each JSON file is named with a UUID (Universally Unique Identifier) in the format {uuid}.json. The UUID is automatically generated (using standard UUID v4) when creating a new evaluation result file. This ensures that:

• Multiple evaluations of the same model can exist without conflicts (each gets a unique UUID)
• Different timestamps are stored as separate files with different UUIDs (not as separate folders)
• A model may have multiple result files, with each file representing different iterations or runs of the leaderboard/evaluation
• UUID's can be generated using Python's uuid.uuid4() function.

Example: The model openai/gpt-4o-2024-11-20 might have multiple files like:

• e70acf51-30ef-4c20-b7cc-51704d114d70.json (evaluation run #1)
• a1b2c3d4-5678-90ab-cdef-1234567890ab.json (evaluation run #2)