{"data":{"slug":"evalplus-evalplus","name":"evalplus","tagline":"Rigorous evaluation of LLM-synthesized code","github_url":"https://github.com/evalplus/evalplus","owner":"evalplus","repo":"evalplus","owner_avatar_url":"https://avatars.githubusercontent.com/u/132106461?v=4","primary_language":"Python","stars":1794,"forks":205,"topics":["benchmark","chatgpt","efficiency","gpt-4","large-language-models","program-synthesis","testing"],"archived":false,"github_pushed_at":"2025-10-02T22:56:38+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/evalplus-evalplus","markdown_url":"https://www.graphcanon.com/tools/evalplus-evalplus.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evalplus-evalplus","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evalplus-evalplus","description":"Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024","homepage_url":"https://evalplus.github.io","license":"Apache-2.0","open_issues":71,"watchers":9,"ai_summary":"Provides tools for benchmarking and evaluating large language models such as GPT-4 and ChatGPT in synthesizing program code.","readme_excerpt":"# Or `pip install \"evalplus[vllm]\" --upgrade` for the latest stable release\n\nevalplus.evaluate --model \"ise-uiuc/Magicoder-S-DS-6.7B\" \\\n                  --dataset [humaneval|mbpp]             \\\n                  --backend vllm                         \\\n                  --greedy\n```\n\n<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>\n<div>\n\n```bash\n\n---\n\n# Code execution within Docker\ndocker run --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \\\n           evalplus.evaluate --dataset humaneval                                     \\\n           --samples /app/humaneval/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_0.0.jsonl\n```\n\n</div>\n</details>\n\n---\n\n# Or `pip install \"evalplus[perf,vllm]\" --upgrade` for the latest stable release\n\nsudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf\nevalplus.evalperf --model \"ise-uiuc/Magicoder-S-DS-6.7B\" --backend vllm\n```\n\n<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>\n<div>\n\n```bash\n\n---\n\n# Code execution within Docker\nsudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf\ndocker run --cap-add PERFMON --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \\\n           evalplus.evalperf --samples /app/evalperf/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_1.0.jsonl\n```\n\n</div>\n</details>\n\n---\n\n# Install Flash Attention 2\npip install packaging ninja\npip install flash-attn --no-build-isolation\n\n---\n\n# Note: if you have installation problem, consider using pre-built","github_created_at":"2023-04-15T04:20:10+00:00","created_at":"2026-07-11T23:45:39.037704+00:00","updated_at":"2026-08-05T18:00:59.513725+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"chatgpt","name":"chatgpt"},{"slug":"efficiency","name":"efficiency"},{"slug":"program-synthesis","name":"program-synthesis"},{"slug":"testing","name":"testing"}],"trust":{"provenance":{"is_fork":false,"github_id":628154495,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-05T18:00:57.948Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":306,"last_release_at":"2024-10-20T21:59:03Z"},"security_summary":{"status":"ok","scanner":"osv@v1","low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:45:42.570Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-05T18:00:58.975Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-05T18:00:58.975Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-05T18:00:58.975Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-05T18:00:58.975Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-05T18:00:58.975Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need advanced benchmarks specific to large language model generated program synthesis, such as from GPT-4 or ChatGPT.","For projects requiring safe execution of synthesized code inside Docker containers with perf monitoring enabled.","If your development involves Magicoder-S-DS-6.7B and similar models where precise evaluation tools are crucial."],"when_not_to_use":["Avoid if you require broad language model benchmarking that is not specifically focused on program synthesis capabilities.","Do not use evalplus if your project does not benefit from or need Docker-based isolation for code execution safety measures, such as in controlled lab environments without external dependencies."],"source":"enrich:decision_facts","observed_at":"2026-07-16T19:34:41.467Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"evalplus offers tools for rigorously benchmarking and evaluating large language models like GPT-4 and ChatGPT in synthesizing program code using Python under the Apache-2.0 license."}]}}