Home/Evaluation & Observability/code-eval

code-eval

abacaj/code-eval

Run evaluation on LLMs using human-eval benchmark.

GitHub

GraphCanon updated Oct 6, 2026 · GitHub synced Oct 6, 2026

57views this month

432 stars36 forksLast push Sep 12, 2023 Python MIT

Decision brief

code-eval assesses large language models with the human-eval benchmark to provide insights into code generation reliability.

Good fit when

When you need clear comparisons of pass rates for different LLMs using standardized tests
If your project involves optimizing for edge cases in model tokenizing and loading processes

Avoid when

If the tool's results do not correlate well with the official published benchmarks due to unknown prompt differences
For real-time or dynamic evaluations as this repo offers pre-computed static results only

Observed Jul 17, 2026 · Source: enrich:decision_facts

code-eval alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Dormant (1120d since push); As of Oct 6, 2026
Provenance: Not a fork · Personal account; As of Oct 6, 2026
Security (OSV): 73 low (73 low); As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install code-eval

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

deepeval

LLM Evaluation Framework.

18k★

lm-evaluation-harness

A framework for few-shot evaluation of language models.

14k★

human-eval

Evaluating Large Language Models Trained on Code

3.4k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

A Python-based repository to evaluate the performance of large language models using the human-eval benchmark. It assesses different models and presents results based on their pass rates at various points, offering insights into model reliability in generating code solutions.

Capability facts

Languages: python
Source: github.language · Oct 6, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Oct 6, 2026)

Create python environment

Source link

README

code eval What This is a repo I use to run human eval on code models, adjust as needed. Some scripts were adjusted from wizardcoder repo (process eval.py). The evaluation code is duplicated in several files, mostly to handle edge cases around model tokenizing and loading (will cl...

For agents

This page has a .md twin and JSON over the API.

/tools/abacaj-code-eval.md GET /api/graphcanon/tools/abacaj-code-eval GET /api/graphcanon/graph?tool=abacaj-code-eval