{"data":{"slug":"bigcode-project-bigcode-evaluation-harness","name":"bigcode-evaluation-harness","tagline":"A framework for the evaluation of autoregressive code generation language models.","github_url":"https://github.com/bigcode-project/bigcode-evaluation-harness","owner":"bigcode-project","repo":"bigcode-evaluation-harness","owner_avatar_url":"https://avatars.githubusercontent.com/u/110470554?v=4","primary_language":"Python","stars":1053,"forks":263,"topics":[],"archived":false,"github_pushed_at":"2025-07-22T13:18:09+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/bigcode-project-bigcode-evaluation-harness","markdown_url":"https://www.graphcanon.com/tools/bigcode-project-bigcode-evaluation-harness.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bigcode-project-bigcode-evaluation-harness","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bigcode-project-bigcode-evaluation-harness","description":"A framework for the evaluation of autoregressive code generation language models.","homepage_url":null,"license":"Apache-2.0","open_issues":97,"watchers":10,"ai_summary":null,"readme_excerpt":"## Docker containers\nFor safety, we provide a Dockerfiles to do the execution inside a docker container. To do that, first, do the generation on your machine and save them in `generations.json` for example by adding the flag `--generation_only` to the command. Then use the Docker image that we provide:\n\n```bash\n$ docker pull ghcr.io/bigcode-project/evaluation-harness\n$ docker tag ghcr.io/bigcode-project/evaluation-harness evaluation-harness\n```\n\nIf you want to evaluate on MultiPL-E, we have a different Dockerfile since it requires more dependencies, use:\n```bash\n$ docker pull ghcr.io/bigcode-project/evaluation-harness-multiple\n$ docker tag ghcr.io/bigcode-project/evaluation-harness-multiple evaluation-harness-multiple\n```\n\n---\n\n### Building  Docker images\n\nIf you modify the evaluation harness, you may want to rebuild the docker images.\n\nHere's how to build a docker image for the evaluation harness:\n```bash\n$ sudo make DOCKERFILE=Dockerfile  all\n```\nThis creates an image called `evaluation-harness`, and runs a test on it. To skip the test remove `all` form the command.\n\nFor MultiPL-E:\n```bash\n$ sudo make DOCKERFILE=Dockerfile-multiple all\n```\nThis creates an image called `evaluation-harness-multiple`.","github_created_at":"2022-08-09T12:58:56+00:00","created_at":"2026-07-11T23:43:29.760136+00:00","updated_at":"2026-07-11T23:43:34.946811+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":522950827,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:43:30.865Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":354,"last_release_at":"2023-05-25T09:37:24Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":46,"high_count":0,"last_scan_at":"2026-07-11T23:43:31.208Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:43:30.639Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-07-11T23:43:30.639Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:43:30.639Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-07-11T23:43:30.639Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:43:30.639Z"}}}}