Home/LLM Frameworks/evalplus

evalplus

Enrichment pending

evalplus/evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

GitHub Website

GraphCanon updated today · GitHub synced today

1.8k

Stars

202

Forks

Open issues

Watchers

9mo

Last push

Python Apache-2.0Created Apr 15, 2023

Trust & integrity

Full report

Maintenance: Slowing (282d since push); As of today · Source: github_public_v1
Provenance: Not a fork · Organization account; As of today · Source: github_public_v1
Security (OSV): No criticals; As of today · Source: osv@v1

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Overview

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

Capability facts

Deploy: Self-host
Source: dockerfile:Dockerfile · Jul 11, 2026
Docker: Dockerfile present
Source: dockerfile:Dockerfile · Jul 11, 2026
Languages: python
Source: github.language+pyproject.toml · Jul 11, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Jul 11, 2026)

# Or `pip install "evalplus[vllm]" --upgrade` for the latest stable release

Source link

README

Or `pip install "evalplus[vllm]" --upgrade` for the latest stable release

evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B"
--dataset [humaneval|mbpp]
--backend vllm
--greedy


<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>
<div>

```bash

---

# Code execution within Docker
docker run --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \
           evalplus.evaluate --dataset humaneval                                     \
           --samples /app/humaneval/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_0.0.jsonl

Or `pip install "evalplus[perf,vllm]" --upgrade` for the latest stable release

sudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf evalplus.evalperf --model "ise-uiuc/Magicoder-S-DS-6.7B" --backend vllm


<details><summary>🛡️ Safe code execution within Docker <i>:: click to expand ::</i></summary>
<div>

```bash

---

# Code execution within Docker
sudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf
docker run --cap-add PERFMON --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \
           evalplus.evalperf --samples /app/evalperf/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_1.0.jsonl