Quick Start

This project uses uv with a checked-in uv.lock for reproducible builds. To get started, sync the repo with uv, browse the available tasks and suites, and preview a run with the built-in mock provider.

Install uv if not already installed

curl -LsSf https://astral.sh/uv/install.sh | sh

Install Python 3.12 if your machine does not already have it

uv python install 3.12

Install dependencies + the package (editable) from the lockfile.

Quick Start: Minimal Task Example

"""Example: Minimal task implementation."""
from collections.abc import Iterator
from typing import Any

from olmo_eval.common.types import Instance, LMOutput, LMRequest, RequestType
from olmo_eval.data import DataLoader, DataSource
from olmo_eval.evals.tasks.common import Task, register


@register("my_task")
class MyTask(Task):
    """My task implementation."""

    # DataSource arguments:
    #   path: HuggingFace dataset path (e.g., "cais/mmlu")
    #   subset: Dataset subset/config (e.g., "abstract_algebra")
    #   split: Dataset split (e.g., "test", "validation")
    data_source = DataSource(path="cais/mmlu", subset="abstract_algebra", split="test")

    @property
    def instances(self) -> Iterator[Instance]:
        """Load and yield instances from the dataset."""
        if self._instances_cache is None:
            self._instances_cache = []
            loader = DataLoader()
            source = self.config.get_data_source()
            for doc in loader.load(source):
                self._instances_cache.append(self.process_doc(doc))
        yield from self._instances_cache

    def process_doc(self, doc: dict[str, Any]) -> Instance:
        """Convert a dataset document to an Instance."""
        return Instance(
            question=doc["question"],
            gold_answer=doc["answer"],
            choices=tuple(doc["choices"]),  # For MC tasks
            metadata={"id": doc["id"]},
        )

    def format_request(self, instance: Instance) -> LMRequest:
        """Format instance for the language model."""
        if self.config.formatter is not None:
            return self.config.formatter.format(instance, self.get_fewshot())
        # Fallback formatting
        return LMRequest(request_type=RequestType.COMPLETION, prompt=instance.question)

    def extract_answer(self, output: LMOutput) -> str | None:
        """Extract the answer from model output."""
        return output.text.strip()

Installation

The beaker extra is included in the default dev group, so a plain uv sync --frozen is enough. If you previously opted out of the default groups, re-enable it with:

uv sync --frozen --extra beaker

Docker Image Management

Docker images provide the runtime environment (Python, PyTorch, CUDA) but do NOT include:

• Source code - Gantry mounts your git repository at runtime
• Inference providers - Installed at job startup from each model's resolved provider config

This approach allows you to:

• Use any git commit without rebuilding images
• Keep images small and cacheable

Manual installation inside container

uv pip install -e '.[vllm]' # includes vllm[runai]

For raw OLMo-core checkpoints, force the provider kind through the harness and
use `provider.package` when you need a specific `ai2-olmo-core` install:

```bash

---

# Install dependencies from the lockfile
uv sync --frozen