GraphCanon updated 2w · GitHub synced 2w · 25 views this month
Decision brief
human-eval is a tool designed for evaluating large language models trained specifically on code through human-written tests.
Good fit when
- When you need to evaluate the performance of AI systems that have been trained exclusively on code datasets, as it allows testing via human-created benchmarks relevant only to code-based models.
- For researchers and developers aiming to assess how well a language model understands programming syntax and semantics.
Avoid when
- If you are interested in evaluating general natural language processing tasks without coding context, as human-eval is tailored specifically for assessing code-focused AI systems.
- When the required Python version is below 3.7; this tool mandates at least Python 3.7 to ensure compatibility with its dependencies.
- Hosting:
- self hosted - This evaluation framework must be installed and set up in your own environment, ensuring full control over the testing process.
- Pricing:
- freemium - The software is available under an MIT license for free use, yet advanced features or services beyond its core functionality might incur costs.
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (564d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No criticals
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for OpenAI. Display-only - separate from trust and ranking.
- Company
- OpenAI·GitHub org profile·1mo
- Employees
- 4,500·Wikidata (P1128 employees)·1mo
- Funding
- $13,000,000,000 (2023-01)·GraphCanon curated seed (public press)·1mo
- Commercial model
- OSS + managed cloud·GraphCanon curated seed·1mo
Install
pip install human-evalSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Repository for evaluating large language models trained on code using human-written tests.
Capability facts
- Languages
- python
Source: github.language · Aug 5, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 5, 2026)
Make sure to use python 3.7 or later:Source link
Tags
README
Installation
Make sure to use python 3.7 or later:
$ conda create -n codex python=3.7
$ conda activate codex
Check out and install this repository:
$ git clone https://github.com/openai/human-eval
$ pip install -e human-eval
For agents
This page has a .md twin and JSON over the API.