{"data":{"slug":"abacaj-code-eval","name":"code-eval","tagline":"Run evaluation on LLMs using human-eval benchmark.","github_url":"https://github.com/abacaj/code-eval","owner":"abacaj","repo":"code-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/7272343?v=4","primary_language":"Python","stars":431,"forks":37,"topics":["humaneval","wizardcoder"],"archived":false,"github_pushed_at":"2023-09-12T03:03:56+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/abacaj-code-eval","markdown_url":"https://www.graphcanon.com/tools/abacaj-code-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/abacaj-code-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=abacaj-code-eval","description":"Run evaluation on LLMs using human-eval benchmark","homepage_url":null,"license":"MIT","open_issues":5,"watchers":9,"ai_summary":"A Python-based repository to evaluate the performance of large language models using the human-eval benchmark. It assesses different models and presents results based on their pass rates at various points, offering insights into model reliability in generating code solutions.","readme_excerpt":"# code-eval\n\n## What\n\nThis is a repo I use to run human-eval on code models, adjust as needed. Some scripts were adjusted from wizardcoder repo ([process_eval.py](https://github.com/nlpxucan/WizardLM/blob/main/WizardCoder/src/process_humaneval.py)). The evaluation code is duplicated in several files, mostly to handle edge cases around model tokenizing and loading (will clean it up).\n\n## Results\n\nTable is sorted by pass@1 score.\n \n| model                                                                                                 | size | pass@1  | pass@10 | screenshot                                                                                                         |\n| ----------------------------------------------------------------------------------------------------- | ---- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------ |\n| [sahil2801/replit-code-instruct-glaive](https://huggingface.co/sahil2801/replit-code-instruct-glaive) | 3B   | 63.5%   | 67%     |         |\n| [WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0)                          | 15B  | 57%     | 68.9%   |             |\n| [bigcode/starcoder](https://huggingface.co/bigcode/starcoder)                                         | 15B  | 34.6%   | 48.7%   |               |\n| [openchat/opencoderplus](https://huggingface.co/openchat/opencoderplus)                               | 15B  | 27.3%   | 43.9%   |               |\n| [teknium/Replit-v1-CodeInstruct-3B](https://huggingface.co/teknium/Replit-v1-CodeInstruct-3B)         | 3B   | 25.8%   | 42.6%   |  |\n| [teknium/Replit-v2-CodeInstruct-3B](https://huggingface.co/teknium/Replit-v2-CodeInstruct-3B)         | 3B   | 21.5%   | 31%     |  |\n| [replit-code-v1-3b](https://huggingface.co/replit/replit-code-v1-3b)                                  | 3B   | 17.1%   | 29.8%   |          |\n| [mpt-7b](https://huggingface.co/mosaicml/mpt-7b)                                                      | 7B   | 15.9%   | 23.7%   |                  |\n| [xgen-7b-8k-base](https://huggingface.co/Salesforce/xgen-7b-8k-base)                                  | 7B   | 14.9%   | 22.5%   |         |\n| [openllama-7b-v2](https://huggingface.co/openlm-research/open_llama_7b)                               | 7B   | 14%     | 23.1%   |         |\n| [llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf)                                                | 7B   | 13.1%   | 21.9%   |                                                                                                |\n| [llama-7b](https://huggingface.co/huggyllama/llama-7b)                                                | 7B   | 12.1%   | 18.9%   |                                                                                                |\n| [mpt-30b](https://huggingface.co/mosaicml/mpt-30b)                                                    | 30B  | pending | pending | pending                                                                                                            |\n\n## FAQ\n\n> Why is there a discrepancy on some of the scores between official numbers? \n\nBecause it is not obvious or published what prompt or processing the official models used to conduct their evaluation on this benchmark. The goal here is to try and best reproduce those numbers, in many cases it is possible to get very close to the published numbers.\n\nAll of the scores here were run independently of any published numbers and are reproducible by cloning the repo and following the setup.\n\n> Why do some models have a filter_code post generation step?\n\nBase models can in many cases repeat outputs, breaking the benchmark scores. Instruct models don't have this problem and so you won't see this step, they tend to output a end of sequence token.\n\n## Setup\n\nCreate python environment\n\n```sh\npython -m venv env && source env/bin/activate\n```\n\nInstall dependencies\n\n```sh\npip install -r requirements.txt","github_created_at":"2023-07-01T22:16:02+00:00","created_at":"2026-07-11T23:43:33.494191+00:00","updated_at":"2026-08-05T12:01:48.467564+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"humaneval","name":"humaneval"},{"slug":"wizardcoder","name":"wizardcoder"}],"trust":{"provenance":{"is_fork":false,"github_id":661130724,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-05T12:01:47.477Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1058,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":73,"high_count":0,"last_scan_at":"2026-07-11T23:43:41.595Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-05T12:01:47.972Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-05T12:01:47.972Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-05T12:01:47.972Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need clear comparisons of pass rates for different LLMs using standardized tests","If your project involves optimizing for edge cases in model tokenizing and loading processes"],"when_not_to_use":["If the tool's results do not correlate well with the official published benchmarks due to unknown prompt differences","For real-time or dynamic evaluations as this repo offers pre-computed static results only"],"source":"enrich:decision_facts","observed_at":"2026-07-17T12:48:50.321Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"code-eval assesses large language models with the human-eval benchmark to provide insights into code generation reliability."}]}}