{"data":{"slug":"evaleval-every-eval-ever","name":"every_eval_ever","tagline":"Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results, from leaderboard scrapes and research papers to local ev","github_url":"https://github.com/evaleval/every_eval_ever","owner":"evaleval","repo":"every_eval_ever","owner_avatar_url":"https://avatars.githubusercontent.com/u/176316740?v=4","primary_language":"Python","stars":93,"forks":42,"topics":["agent-evaluation","ai-evaluation","evaluations","infra","llm-evaluation"],"archived":false,"github_pushed_at":"2026-07-04T22:10:45+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/evaleval-every-eval-ever","markdown_url":"https://www.graphcanon.com/tools/evaleval-every-eval-ever.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evaleval-every-eval-ever","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evaleval-every-eval-ever","description":"Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.","homepage_url":"https://evalevalai.com/projects/every-eval-ever/","license":"MIT","open_issues":48,"watchers":3,"ai_summary":null,"readme_excerpt":"# Every Eval Ever\n\n> [EvalEval Coalition](https://evalevalai.com) — \"We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations.\"\n\n📖 **[Documentation](https://evalevalai.com/every_eval_ever/)**\n\n**Every Eval Ever** is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused. The three components that make it work:\n\n- 📋 **A metadata schema** ([`eval.schema.json`](eval.schema.json)) that defines the information needed for meaningful comparison of evaluation results, including [instance-level data](instance_level_eval.schema.json)\n- 🔧 **Validation** that checks data against the schema before it enters the repository\n- 🔌 **Converters** for [Inspect AI](every_eval_ever/converters/inspect/), [HELM](every_eval_ever/converters/helm/), and [lm-eval-harness](every_eval_ever/converters/lm_eval/), so you can transform your existing evaluation logs into the standard format\n\nInstall the package:\n\n```bash\npip install every-eval-ever\n```\n\nOptional converter dependencies:\n\n```bash\npip install 'every-eval-ever[inspect]'\npip install 'every-eval-ever[helm]'\npip install 'every-eval-ever[all]'\n```\n\n### Terminology\n\n| Term | Our Definition | Example |\n|---|---|---|\n| **Single Benchmark** | Standardized eval using one dataset to test a single capability, producing one score | MMLU — ~15k multiple-choice QA across 57 subjects |\n| **Composite Benchmark** | A collection of simple benchmarks aggregated into one overall score, testing multiple capabilities at once | BIG-Bench bundles >200 tasks with a single aggregate score |\n| **Metric** | Any numerical or categorical value used to score performance on a benchmark (accuracy, F1, precision, recall, …) | A model scores 92% accuracy on MMLU |\n\n## 🚀 Contributor Guide\nNew data can be contributed to the [Hugging Face Dataset](https://huggingface.co/datasets/evaleval/EEE_datastore) using the following process:\n\nLeaderboard/evaluation data is split-up into files by individual model, and data for each model is stored using [`eval.schema.json`](eval.schema.json). The repository is structured into folders as `data/{benchmark_name}/{developer_name}/{model_name}/`.\n\n### TL;DR How to successfully submit\n\n1. Data must conform to [`eval.schema.json`](eval.schema.json) (current version: `0.2.2`)\n2. The validation pipeline will automatically verify the data submitted in the pull request, but can also be manually triggered by typing ```/eee validate changed``` in a comment on the HF PR.\n3. An EvalEval member will review and merge your submission\n\n### PR Naming Convention\n\nUse these prefixes in your pull request titles:\n\n- `[Submission]` - New evaluation data\n- `[Issue #N]` - Fix for a specific GitHub issue\n- `[Feature]` - New functionality not tied to an issue\n- `[Docs]` - Documentation changes\n- `[ACL Shared Task]` - Shared task submissions (priority review)\n\n### UUID Naming Convention\n\nEach JSON file is named with a **UUID (Universally Unique Identifier)** in the format `{uuid}.json`. The UUID is automatically generated (using standard UUID v4) when creating a new evaluation result file. This ensures that:\n- **Multiple evaluations** of the same model can exist without conflicts (each gets a unique UUID)\n- **Different timestamps** are stored as separate files with different UUIDs (not as separate folders)\n- A model may have multiple result files, with each file representing different iterations or runs of the leaderboard/evaluation\n- UUID's can be generated using Python's `uuid.uuid4()` function.\n\n**Example**: The model `openai/gpt-4o-2024-11-20` might have multiple files like:\n- `e70acf51-30ef-4c20-b7cc-51704d114d70.json` (evaluation run #1)\n- `a1b2c3d4-5678-90ab-cdef-1234567890ab.json` (evaluation run #2)","github_created_at":"2025-10-08T17:35:36+00:00","created_at":"2026-07-15T10:39:15.98368+00:00","updated_at":"2026-07-15T10:39:18.903764+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"agent-evaluation","name":"agent-evaluation"},{"slug":"ai-evaluation","name":"ai-evaluation"},{"slug":"evaluations","name":"evaluations"},{"slug":"infra","name":"infra"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":1072418796,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-15T10:39:16.948Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":1,"days_since_push":10,"last_release_at":"2026-06-01T22:00:10Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T10:39:17.365Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-15T10:39:16.710Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-15T10:39:16.710Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-15T10:39:16.710Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-15T10:39:16.710Z"}}}}