{"data":{"slug":"evaleval-every-eval-ever","name":"every_eval_ever","tagline":"Shared schema and crowdsourced eval database","github_url":"https://github.com/evaleval/every_eval_ever","owner":"evaleval","repo":"every_eval_ever","owner_avatar_url":"https://avatars.githubusercontent.com/u/176316740?v=4","primary_language":"Python","stars":111,"forks":49,"topics":["agent-evaluation","ai-evaluation","evaluations","infra","llm-evaluation"],"archived":false,"github_pushed_at":"2026-09-07T11:57:22+00:00","maintenance_label":"Very active","stars_delta_30d":9,"url":"https://www.graphcanon.com/tools/evaleval-every-eval-ever","markdown_url":"https://www.graphcanon.com/tools/evaleval-every-eval-ever.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evaleval-every-eval-ever","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evaleval-every-eval-ever","description":"Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.","homepage_url":"https://evalevalai.com/projects/every-eval-ever/","license":"MIT","open_issues":27,"watchers":3,"ai_summary":"Every Eval Ever defines a standardized metadata format for storing AI evaluation results from various sources including leaderboard scrapes, research papers, and local runs.","readme_excerpt":"# Every Eval Ever\n\n> [EvalEval Coalition](https://evalevalai.com) — \"We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations.\"\n\n📄 **[Paper (arXiv:2606.14516)](https://arxiv.org/abs/2606.14516)**\n\n**Every Eval Ever** is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused. The three components that make it work:\n\n- 📋 **A metadata schema** ([`eval.schema.json`](every_eval_ever/schemas/eval.schema.json)) that defines the information needed for meaningful comparison of evaluation results, including [instance-level data](every_eval_ever/schemas/instance_level_eval.schema.json)\n- 🔧 **Validation** that checks data against the schema before it enters the repository\n- 🔌 **Converters** for [Inspect AI](every_eval_ever/converters/inspect/), [HELM](every_eval_ever/converters/helm/), and [lm-eval-harness](every_eval_ever/converters/lm_eval/), so you can transform your existing evaluation logs into the standard format\n\nAdd the package to your project:\n\n```bash\nuv add every-eval-ever\n```\n\nOptional converter dependencies:\n\n```bash\nuv add 'every-eval-ever[inspect]'\nuv add 'every-eval-ever[helm]'\nuv add 'every-eval-ever[all]'\n```\n\n> [!NOTE]\n> **`helm` extra + nltk's import guard.** The `helm` extra pulls in `nltk`, and\n> nltk ≥ 3.10.1 ships an import guard (`nltk/inisec.py`, a CWE-427 mitigation)\n> that blocks nltk-initiated imports of any module whose file resolves *under the\n> current working directory*. With the common in-project virtualenv layout\n> (`uv`'s `.venv/`, or any venv inside the repo), site-packages sits under the\n> CWD, so the guard trips on nltk's own dependencies and importing the HELM\n> converter fails. The extra currently caps `nltk<3.10.1`, so this does not bite\n> by default. If you move to a newer nltk, keep the environment **outside** the\n> checkout — the same thing CI does via `UV_PROJECT_ENVIRONMENT` — e.g.\n> `UV_PROJECT_ENVIRONMENT=/tmp/eee-venv uv sync --extra helm`, or create your\n> venv outside the repository. Upstream tracker:\n> [nltk#3730](https://github.com/nltk/nltk/issues/3730).\n\n### Terminology\n\n| Term | Our Definition | Example |\n|---|---|---|\n| **Single Benchmark** | Standardized eval using one dataset to test a single capability, producing one score | MMLU — ~15k multiple-choice QA across 57 subjects |\n| **Composite Benchmark** | A collection of simple benchmarks aggregated into one overall score, testing multiple capabilities at once | BIG-Bench bundles >200 tasks with a single aggregate score |\n| **Metric** | Any numerical or categorical value used to score performance on a benchmark (accuracy, F1, precision, recall, …) | A model scores 92% accuracy on MMLU |\n\n## 🚀 Contributing\n\nContributing data, writing an adapter, or changing the schema? **[CONTRIBUTING.md](CONTRIBUTING.md)** is the guide: how your PR gets reviewed, how to submit data to the datastore, the naming conventions, and the conventions for filling in each field.\n\nThe rest of this README is for *reading* EEE — what the schema says, how to validate a record, and what the converters do.\n\n## 🧩 Instance-Level Data\n\nFor evaluations that include per-sample results, the individual results should be stored in a companion `{uuid}_samples.jsonl` file in the same folder (one JSONL per JSON, sharing the same UUID). The aggregate JSON file refers to its JSONL via the `detailed_evaluation_results` field. The instance-level schema ([`instance_level_eval.schema.json`](every_eval_ever/schemas/instance_level_eval.schema.json)) supports three interaction types:\n\n- **`single_turn`**: Standard QA, MCQ, classification — uses `output` object\n- **`multi_turn`**: Conversational evaluations with multiple exchanges — uses `messages` array\n- **`agenti","github_created_at":"2025-10-08T17:35:36+00:00","created_at":"2026-07-15T10:39:15.98368+00:00","updated_at":"2026-09-20T04:24:49.379227+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agent-evaluation","name":"agent-evaluation"},{"slug":"ai-evaluation","name":"ai-evaluation"},{"slug":"evaluations","name":"evaluations"},{"slug":"infra","name":"infra"},{"slug":"llm-evaluation","name":"llm-evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":1072418796,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-09T06:00:06.538Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1,"last_release_at":"2026-06-01T22:00:10Z","stars_delta_30d":9,"open_issues_delta_30d":3},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T10:39:17.365Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-09T06:00:07.323Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-09T06:00:07.323Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-09T06:00:07.323Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-09T06:00:07.323Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"Every Eval Ever is open-source under the MIT license, allowing free use and modification. No direct costs are associated with using the schema or contributing to the database."},"requirements":{"notes":["To utilize all features, you need to install specific converter dependencies via pip."],"min_ram_gb":2,"requires_docker":false},"constraints":{"min_ram_gb":2,"pricing_model":"freemium","requires_docker":false},"when_to_use":["Use Every Eval Ever if you need to compare evaluation results from different frameworks in a consistent manner, ensuring results can be easily reproduced or reused as they conform to a defined schema.","Opt for this project when your workflow involves leaderboard scrapes and research paper data that you want to integrate with local runs of AI models."],"when_not_to_use":["Avoid Every Eval Ever if you require real-time updates on evaluation results, as the database relies on contributions from a community to maintain and update its dataset.","If your project needs to integrate evaluation outcomes without an explicit need for extensive metadata validation or standardization, this tool might be less suitable."],"source":"enrich:decision_facts","observed_at":"2026-07-16T20:45:43.734Z"},"constraint_facets":{"min_ram_gb":2,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium - Every Eval Ever is open-source under the MIT license, allowing free use and modification. No direct costs are associated with using the schema or contributing to the database."},{"label":"Requirements","value":"Min 2 GB RAM; To utilize all features, you need to install specific converter dependencies via pip."},{"label":"Adopt for","value":"Every Eval Ever is dedicated to providing a standardized metadata framework and a crowdsourced evaluation database for AI results."}]}}