{"data":{"slug":"allenai-olmo-eval","name":"olmo-eval","tagline":"olmo-eval","github_url":"https://github.com/allenai/olmo-eval","owner":"allenai","repo":"olmo-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/5667695?v=4","primary_language":"Python","stars":60,"forks":10,"topics":[],"archived":false,"github_pushed_at":"2026-07-11T03:46:50+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/allenai-olmo-eval","markdown_url":"https://www.graphcanon.com/tools/allenai-olmo-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/allenai-olmo-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=allenai-olmo-eval","description":null,"homepage_url":null,"license":"Apache-2.0","open_issues":32,"watchers":0,"ai_summary":null,"readme_excerpt":"## Quick Start\n\nThis project uses [uv](https://docs.astral.sh/uv/) with a checked-in `uv.lock`\nfor reproducible builds. To get started, sync the repo with `uv`, browse the\navailable tasks and suites, and preview a run with the built-in `mock` provider.\n\n---\n\n# Install uv if not already installed\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n---\n\n# Install Python 3.12 if your machine does not already have it\nuv python install 3.12\n\n---\n\n# Install dependencies + the package (editable) from the lockfile.\n\n---\n\n### Quick Start: Minimal Task Example\n\n```python\n\"\"\"Example: Minimal task implementation.\"\"\"\nfrom collections.abc import Iterator\nfrom typing import Any\n\nfrom olmo_eval.common.types import Instance, LMOutput, LMRequest, RequestType\nfrom olmo_eval.data import DataLoader, DataSource\nfrom olmo_eval.evals.tasks.common import Task, register\n\n\n@register(\"my_task\")\nclass MyTask(Task):\n    \"\"\"My task implementation.\"\"\"\n\n    # DataSource arguments:\n    #   path: HuggingFace dataset path (e.g., \"cais/mmlu\")\n    #   subset: Dataset subset/config (e.g., \"abstract_algebra\")\n    #   split: Dataset split (e.g., \"test\", \"validation\")\n    data_source = DataSource(path=\"cais/mmlu\", subset=\"abstract_algebra\", split=\"test\")\n\n    @property\n    def instances(self) -> Iterator[Instance]:\n        \"\"\"Load and yield instances from the dataset.\"\"\"\n        if self._instances_cache is None:\n            self._instances_cache = []\n            loader = DataLoader()\n            source = self.config.get_data_source()\n            for doc in loader.load(source):\n                self._instances_cache.append(self.process_doc(doc))\n        yield from self._instances_cache\n\n    def process_doc(self, doc: dict[str, Any]) -> Instance:\n        \"\"\"Convert a dataset document to an Instance.\"\"\"\n        return Instance(\n            question=doc[\"question\"],\n            gold_answer=doc[\"answer\"],\n            choices=tuple(doc[\"choices\"]),  # For MC tasks\n            metadata={\"id\": doc[\"id\"]},\n        )\n\n    def format_request(self, instance: Instance) -> LMRequest:\n        \"\"\"Format instance for the language model.\"\"\"\n        if self.config.formatter is not None:\n            return self.config.formatter.format(instance, self.get_fewshot())\n        # Fallback formatting\n        return LMRequest(request_type=RequestType.COMPLETION, prompt=instance.question)\n\n    def extract_answer(self, output: LMOutput) -> str | None:\n        \"\"\"Extract the answer from model output.\"\"\"\n        return output.text.strip()\n```\n\n---\n\n### Installation\n\nThe `beaker` extra is included in the default `dev` group, so a plain\n`uv sync --frozen` is enough. If you previously opted out of the default\ngroups, re-enable it with:\n\n```bash\nuv sync --frozen --extra beaker\n```\n\n---\n\n## Docker Image Management\n\nDocker images provide the runtime environment (Python, PyTorch, CUDA) but do NOT include:\n- **Source code** - Gantry mounts your git repository at runtime\n- **Inference providers** - Installed at job startup from each model's resolved provider config\n\nThis approach allows you to:\n- Use any git commit without rebuilding images\n- Keep images small and cacheable\n\n---\n\n# Manual installation inside container\nuv pip install -e '.[vllm]'  # includes vllm[runai]\n```\n\nFor raw OLMo-core checkpoints, force the provider kind through the harness and\nuse `provider.package` when you need a specific `ai2-olmo-core` install:\n\n```bash\n\n---\n\n# Install dependencies from the lockfile\nuv sync --frozen","github_created_at":"2026-01-08T18:28:43+00:00","created_at":"2026-07-11T10:34:50.611764+00:00","updated_at":"2026-07-11T10:34:59.103913+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":1130601061,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:34:51.203Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":0,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:56.838Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:34:56.454Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-07-11T10:34:56.454Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T10:34:56.454Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T10:34:56.454Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-07-11T10:34:56.454Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T10:34:56.454Z"}}}}