{"data":{"slug":"bigcode-project-bigcode-evaluation-harness","name":"bigcode-evaluation-harness","tagline":"A framework for evaluating autoregressive code generation language models.","github_url":"https://github.com/bigcode-project/bigcode-evaluation-harness","owner":"bigcode-project","repo":"bigcode-evaluation-harness","owner_avatar_url":"https://avatars.githubusercontent.com/u/110470554?v=4","primary_language":"Python","stars":1055,"forks":261,"topics":[],"archived":false,"github_pushed_at":"2025-07-22T13:18:09+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/bigcode-project-bigcode-evaluation-harness","markdown_url":"https://www.graphcanon.com/tools/bigcode-project-bigcode-evaluation-harness.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bigcode-project-bigcode-evaluation-harness","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bigcode-project-bigcode-evaluation-harness","description":"A framework for the evaluation of autoregressive code generation language models.","homepage_url":null,"license":"Apache-2.0","open_issues":96,"watchers":10,"ai_summary":"bigcode-project/bigcode-evaluation-harness offers a framework to assess the performance of code generation language models in an autoregressive setup, using Python and Docker containers for execution safety and reproducibility. Users can generate outputs locally before evaluation within isolated Docker environments, with separate images tailored for specific setups like MultiPL-E.","readme_excerpt":"## Docker containers\nFor safety, we provide a Dockerfiles to do the execution inside a docker container. To do that, first, do the generation on your machine and save them in `generations.json` for example by adding the flag `--generation_only` to the command. Then use the Docker image that we provide:\n\n```bash\n$ docker pull ghcr.io/bigcode-project/evaluation-harness\n$ docker tag ghcr.io/bigcode-project/evaluation-harness evaluation-harness\n```\n\nIf you want to evaluate on MultiPL-E, we have a different Dockerfile since it requires more dependencies, use:\n```bash\n$ docker pull ghcr.io/bigcode-project/evaluation-harness-multiple\n$ docker tag ghcr.io/bigcode-project/evaluation-harness-multiple evaluation-harness-multiple\n```\n\n---\n\n### Building  Docker images\n\nIf you modify the evaluation harness, you may want to rebuild the docker images.\n\nHere's how to build a docker image for the evaluation harness:\n```bash\n$ sudo make DOCKERFILE=Dockerfile  all\n```\nThis creates an image called `evaluation-harness`, and runs a test on it. To skip the test remove `all` form the command.\n\nFor MultiPL-E:\n```bash\n$ sudo make DOCKERFILE=Dockerfile-multiple all\n```\nThis creates an image called `evaluation-harness-multiple`.","github_created_at":"2022-08-09T12:58:56+00:00","created_at":"2026-07-11T23:43:29.760136+00:00","updated_at":"2026-08-05T12:01:45.945631+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"autoregressive-models","name":"autoregressive models"},{"slug":"code-generation","name":"code generation"},{"slug":"docker","name":"docker"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":522950827,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-05T12:01:45.110Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":378,"last_release_at":"2023-05-25T09:37:24Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":46,"high_count":0,"last_scan_at":"2026-07-11T23:43:31.208Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-05T12:01:45.607Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-05T12:01:45.607Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-05T12:01:45.607Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-05T12:01:45.607Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-05T12:01:45.607Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Users must have Docker installed to leverage the isolated execution environments for model output evaluation."]},"constraints":null,"when_to_use":["When you need to isolate the evaluation environment from your local development setup, ensuring that no external variables affect the outcomes of model performance assessments.","If you are using MultiPL-E or a similar setup requiring specific dependencies, as bigcode-evaluation-harness offers tailored Docker images for these configurations."],"when_not_to_use":["When you require real-time evaluation without the overhead of generating outputs locally and then evaluating them within isolated environments via Docker.","If your model's evaluation process does not necessitate autoregressive setup or the security features provided by Docker, using bigcode-evaluation-harness might introduce unnecessary complexity."],"source":"enrich:decision_facts","observed_at":"2026-07-17T03:16:47.550Z"},"constraint_facets":null,"decision_summary":[{"label":"Requirements","value":"Users must have Docker installed to leverage the isolated execution environments for model output evaluation."},{"label":"Adopt for","value":"bigcode-evaluation-harness is tailored towards evaluating autoregressive code generation models via Python and Docker containers for secure and reproducible execution environments."},{"label":"License detail","value":"bigcode-evaluation-harness is distributed under the Apache-2.0 license."}]}}