GraphCanon updated 2w · GitHub synced 2w
Decision brief
evalplus offers tools for rigorously benchmarking and evaluating large language models like GPT-4 and ChatGPT in synthesizing program code using Python under the Apache-2.0 license.
Good fit when
- When you need advanced benchmarks specific to large language model generated program synthesis, such as from GPT-4 or ChatGPT.
- For projects requiring safe execution of synthesized code inside Docker containers with perf monitoring enabled.
Avoid when
- Avoid if you require broad language model benchmarking that is not specifically focused on program synthesis capabilities.
- Do not use evalplus if your project does not benefit from or need Docker-based isolation for code execution safety measures, such as in controlled lab environments without external dependencies.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (306d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No criticals
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install evalplusSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides tools for benchmarking and evaluating large language models such as GPT-4 and ChatGPT in synthesizing program code.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 5, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 5, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 5, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 5, 2026)
# Or `pip install "evalplus[vllm]" --upgrade` for the latest stable releaseSource link
Tags
README
Or pip install "evalplus[vllm]" --upgrade for the latest stable release
evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B"
--dataset [humaneval|mbpp]
--backend vllm
--greedy
<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>
<div>
```bash
---
# Code execution within Docker
docker run --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \
evalplus.evaluate --dataset humaneval \
--samples /app/humaneval/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_0.0.jsonl
Or pip install "evalplus[perf,vllm]" --upgrade for the latest stable release
sudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf evalplus.evalperf --model "ise-uiuc/Magicoder-S-DS-6.7B" --backend vllm
<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>
<div>
```bash
---
# Code execution within Docker
sudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf
docker run --cap-add PERFMON --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \
evalplus.evalperf --samples /app/evalperf/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_1.0.jsonl
Install Flash Attention 2
pip install packaging ninja pip install flash-attn --no-build-isolation
Note: if you have installation problem, consider using pre-built
For agents
This page has a .md twin and JSON over the API.