GraphCanon updated Oct 12, 2026 · GitHub synced Oct 12, 2026
34views this month
Decision brief
Olmo-eval is an evaluation framework for large language models, using uv for reproducible builds. It focuses on modular task implementations and integrates with various datasets via defined tasks.
Good fit when
- When you need a flexible evaluation setup that works with a variety of LLMs and datasets.
- For reproducible evaluations where your build environment needs to be consistent across runs.
Avoid when
- When you require a simpler setup that doesn't need the reproducibility constraints of uv builds.
- If your project already has an established evaluation toolchain and does not benefit from introducing a new framework for manageability reasons.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Very active (0d since push)
- As of Oct 12, 2026
- Provenance
- Not a fork · Organization account
- As of Oct 12, 2026
- Security (OSV)
- No lockfile
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install olmo-evalSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A framework to evaluate large language models through various tasks and datasets, using reproducible build setup with uv.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Oct 12, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Oct 12, 2026
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Oct 12, 2026
- Languages
- python
Source: github.language+pyproject.toml · Oct 12, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Oct 12, 2026)
# Install Python 3.12 if your machine does not already have itSource link
Tags
README
Quick Start This project uses uv with a checked in for reproducible builds. To get started, sync the repo with , browse the available tasks and suites, and preview a run with the built in provider. Install uv if not already installed curl LsSf https://astral.sh/uv/install.sh sh I...
For agents
This page has a .md twin and JSON over the API.