bigcode-evaluation-harness
A framework for evaluating autoregressive code generation language models.
GraphCanon updated 2w · GitHub synced 2w
Decision brief
bigcode-evaluation-harness is tailored towards evaluating autoregressive code generation models via Python and Docker containers for secure and reproducible execution environments.
Good fit when
- When you need to isolate the evaluation environment from your local development setup, ensuring that no external variables affect the outcomes of model performance assessments.
- If you are using MultiPL-E or a similar setup requiring specific dependencies, as bigcode-evaluation-harness offers tailored Docker images for these configurations.
Avoid when
- When you require real-time evaluation without the overhead of generating outputs locally and then evaluating them within isolated environments via Docker.
- If your model's evaluation process does not necessitate autoregressive setup or the security features provided by Docker, using bigcode-evaluation-harness might introduce unnecessary complexity.
- Requirements:
- Users must have Docker installed to leverage the isolated execution environments for model output evaluation.
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (378d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- 46 low (46 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install bigcode-evaluation-harnessSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
bigcode-project/bigcode-evaluation-harness offers a framework to assess the performance of code generation language models in an autoregressive setup, using Python and Docker containers for execution safety and reproducibility. Users can generate outputs locally before evaluation within isolated Docker environments, with separate images tailored for specific setups like MultiPL-E.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 5, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 5, 2026
- Languages
- python
Source: github.language · Aug 5, 2026
Categories
Tags
README
Docker containers
For safety, we provide a Dockerfiles to do the execution inside a docker container. To do that, first, do the generation on your machine and save them in generations.json for example by adding the flag --generation_only to the command. Then use the Docker image that we provide:
$ docker pull ghcr.io/bigcode-project/evaluation-harness
$ docker tag ghcr.io/bigcode-project/evaluation-harness evaluation-harness
If you want to evaluate on MultiPL-E, we have a different Dockerfile since it requires more dependencies, use:
$ docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
$ docker tag ghcr.io/bigcode-project/evaluation-harness-multiple evaluation-harness-multiple
Building Docker images
If you modify the evaluation harness, you may want to rebuild the docker images.
Here's how to build a docker image for the evaluation harness:
$ sudo make DOCKERFILE=Dockerfile all
This creates an image called evaluation-harness, and runs a test on it. To skip the test remove all form the command.
For MultiPL-E:
$ sudo make DOCKERFILE=Dockerfile-multiple all
This creates an image called evaluation-harness-multiple.
For agents
This page has a .md twin and JSON over the API.