GraphCanon updated Oct 8, 2026 · GitHub synced Oct 8, 2026
69views this month
Decision brief
evalplus offers tools for rigorously benchmarking and evaluating large language models like GPT-4 and ChatGPT in synthesizing program code using Python under the Apache-2.0 license.
Good fit when
- When you need advanced benchmarks specific to large language model generated program synthesis, such as from GPT-4 or ChatGPT.
- For projects requiring safe execution of synthesized code inside Docker containers with perf monitoring enabled.
Avoid when
- Avoid if you require broad language model benchmarking that is not specifically focused on program synthesis capabilities.
- Do not use evalplus if your project does not benefit from or need Docker-based isolation for code execution safety measures, such as in controlled lab environments without external dependencies.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (370d since push)
- As of Oct 8, 2026
- Provenance
- Not a fork · Organization account
- As of Oct 8, 2026
- Security (OSV)
- No criticals
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install evalplusSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides tools for benchmarking and evaluating large language models such as GPT-4 and ChatGPT in synthesizing program code.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Oct 8, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Oct 8, 2026
- Languages
- python
Source: github.language+pyproject.toml · Oct 8, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Oct 8, 2026)
# Or `pip install "evalplus[vllm]" --upgrade` for the latest stable releaseSource link
Tags
README
Or for the latest stable release evalplus.evaluate model "ise uiuc/Magicoder S DS 6.7B" \ dataset [humaneval mbpp] \ backend vllm \ greedy bash Code execution within Docker docker run rm pull=always v $(pwd)/evalplus results:/app ganler/evalplus:latest \ evalplus.evaluate dataset...
For agents
This page has a .md twin and JSON over the API.