{"data":{"slug":"evalplus-evalplus","name":"evalplus","tagline":"Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024","github_url":"https://github.com/evalplus/evalplus","owner":"evalplus","repo":"evalplus","owner_avatar_url":"https://avatars.githubusercontent.com/u/132106461?v=4","primary_language":"Python","stars":1774,"forks":202,"topics":["benchmark","chatgpt","efficiency","gpt-4","large-language-models","program-synthesis","testing"],"archived":false,"github_pushed_at":"2025-10-02T22:56:38+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/evalplus-evalplus","markdown_url":"https://www.graphcanon.com/tools/evalplus-evalplus.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evalplus-evalplus","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evalplus-evalplus","description":"Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024","homepage_url":"https://evalplus.github.io","license":"Apache-2.0","open_issues":66,"watchers":9,"ai_summary":null,"readme_excerpt":"# Or `pip install \"evalplus[vllm]\" --upgrade` for the latest stable release\n\nevalplus.evaluate --model \"ise-uiuc/Magicoder-S-DS-6.7B\" \\\n                  --dataset [humaneval|mbpp]             \\\n                  --backend vllm                         \\\n                  --greedy\n```\n\n<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>\n<div>\n\n```bash\n\n---\n\n# Code execution within Docker\ndocker run --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \\\n           evalplus.evaluate --dataset humaneval                                     \\\n           --samples /app/humaneval/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_0.0.jsonl\n```\n\n</div>\n</details>\n\n---\n\n# Or `pip install \"evalplus[perf,vllm]\" --upgrade` for the latest stable release\n\nsudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf\nevalplus.evalperf --model \"ise-uiuc/Magicoder-S-DS-6.7B\" --backend vllm\n```\n\n<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>\n<div>\n\n```bash\n\n---\n\n# Code execution within Docker\nsudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf\ndocker run --cap-add PERFMON --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \\\n           evalplus.evalperf --samples /app/evalperf/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_1.0.jsonl\n```\n\n</div>\n</details>\n\n---\n\n# Install Flash Attention 2\npip install packaging ninja\npip install flash-attn --no-build-isolation\n\n---\n\n# Note: if you have installation problem, consider using pre-built","github_created_at":"2023-04-15T04:20:10+00:00","created_at":"2026-07-11T23:45:39.037704+00:00","updated_at":"2026-07-11T23:45:44.750964+00:00","categories":[{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"chatgpt","name":"chatgpt"},{"slug":"efficiency","name":"efficiency"},{"slug":"gpt-4","name":"gpt-4"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"program-synthesis","name":"program-synthesis"},{"slug":"python","name":"python"},{"slug":"testing","name":"testing"}],"trust":{"provenance":{"is_fork":false,"github_id":628154495,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:45:42.075Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":282,"last_release_at":"2024-10-20T21:59:03Z"},"security_summary":{"status":"ok","scanner":"osv@v1","low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:45:42.570Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:45:41.827Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-07-11T23:45:41.827Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T23:45:41.827Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-07-11T23:45:41.827Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:45:41.827Z"}}}}