GraphCanon updated Oct 18, 2026 · GitHub synced Oct 18, 2026
Decision brief
Unverified catalog guidance: Every Eval Ever is dedicated to providing a standardized metadata framework and a crowdsourced evaluation database for AI results.
Good fit when
- Unverified catalog guidance: Use Every Eval Ever if you need to compare evaluation results from different frameworks in a consistent manner, ensuring results can be easily reproduced or reused as they conform to a defined schema.
- Unverified catalog guidance: Opt for this project when your workflow involves leaderboard scrapes and research paper data that you want to integrate with local runs of AI models.
Avoid when
- Unverified catalog guidance: Avoid Every Eval Ever if you require real-time updates on evaluation results, as the database relies on contributions from a community to maintain and update its dataset.
- Unverified catalog guidance: If your project needs to integrate evaluation outcomes without an explicit need for extensive metadata validation or standardization, this tool might be less suitable.
- Pricing:
- freemium - Every Eval Ever is open-source under the MIT license, allowing free use and modification. No direct costs are associated with using the schema or contributing to the database.
- Requirements:
- Min 2 GB RAM; To utilize all features, you need to install specific converter dependencies via pip.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Very active (5d since push)
- As of Oct 18, 2026
- Provenance
- Not a fork · Organization account
- As of Oct 18, 2026
- Security (OSV)
- No lockfile
- As of Jul 15, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
Package installation is not established from the available documentation. Check the source repository for setup instructions.
git clone https://github.com/evaleval/every_eval_everSource repositorySimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Every Eval Ever defines a standardized metadata format for storing AI evaluation results from various sources including leaderboard scrapes, research papers, and local runs.
Capability facts
- CLI
- Declared CLI entrypoint
Source: pyproject.toml:[project.scripts] · Oct 18, 2026
- MCP server
- Unknown; MCP server not established by this scan
Source: repo_scan · Oct 18, 2026
- Languages
- python
Source: github.language+pyproject.toml+.github/ci-types/pyproject.toml · Oct 18, 2026
Categories
Tags
README
Every Eval Ever EvalEval Coalition — "We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations." 📄 Paper (arXiv:2606.14516) Every Eval Ever is a shared schema and crowdsourced eval datab...
For agents
This page has a .md twin and JSON over the API.