GraphCanon updated 2w · GitHub synced 2w
Decision brief
APPS offers a benchmark to evaluate the competence of large language models on coding challenges using its datasets.
Good fit when
- When you need benchmarking datasets specifically tailored for assessing the performance of your AI in solving programming tasks
- For training and evaluating the efficacy of large language models like GPT-2 or GPT-Neo on code generation
Avoid when
- If you solely require general datasets without a focus on coding challenges
- When your use case does not involve using Python-based tools for developing machine learning applications that include program synthesis and code generation
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (777d since push)
- As of 2w
- Provenance
- Not a fork · Personal account
- As of 2w
- Security (OSV)
- 77 low (77 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install appsSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A benchmark for measuring coding challenge competence with datasets and code for training and evaluation using large language models.
Capability facts
- Languages
- python
Source: github.language · Aug 5, 2026
Categories
Tags
README
Measuring Coding Challenge Competence With APPS
This is the repository for Measuring Coding Challenge Competence With APPS by Dan Hendrycks*, Steven Basart*, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.
Download the APPS dataset here. (~1.3GB)
This repository contains both training and evaluation code.
Fine-tuned GPT-2 1.5B and GPT-Neo 2.7B weights are available here.
For other benchmarks of enormous Transformers, see a dataset which tests ability in competition math, a dataset which tests knowledge of ethics, and a dataset spanning 50+ academic subjects.
How to Use
The training instructions are specified in train/README and similarly the evaluation instructions are specified in eval/README.
Hugging Face
The dataset is also available in Hugging Face datasets under apps.
Citation
If you find this useful in your research, please consider citing
@article{hendrycksapps2021,
title={Measuring Coding Challenge Competence With APPS},
author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt},
journal={NeurIPS},
year={2021}
}
For agents
This page has a .md twin and JSON over the API.