{"data":{"slug":"hendrycks-apps","name":"apps","tagline":"APPS: Automated Programming Progress Standard","github_url":"https://github.com/hendrycks/apps","owner":"hendrycks","repo":"apps","owner_avatar_url":"https://avatars.githubusercontent.com/u/11670606?v=4","primary_language":"Python","stars":540,"forks":71,"topics":["code-generation","program-synthesis"],"archived":false,"github_pushed_at":"2024-06-19T06:32:49+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/hendrycks-apps","markdown_url":"https://www.graphcanon.com/tools/hendrycks-apps.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/hendrycks-apps","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=hendrycks-apps","description":"APPS: Automated Programming Progress Standard (NeurIPS 2021)","homepage_url":null,"license":"MIT","open_issues":6,"watchers":11,"ai_summary":"A benchmark for measuring coding challenge competence with datasets and code for training and evaluation using large language models.","readme_excerpt":"# Measuring Coding Challenge Competence With APPS\nThis is the repository for [Measuring Coding Challenge Competence With APPS](https://arxiv.org/pdf/2105.09938) by\n[Dan Hendrycks\\*](https://danhendrycks.com/), [Steven Basart\\*](https://stevenbas.art), [Saurav Kadavath](http://www.sauravkadavath.com), Mantas Mazeika, [Akul Arora](https://github.com/akulaarora), Ethan Guo, [Collin Burns](http://collinpburns.com), Samir Puranik, [Horace He](http://horace.io), [Dawn Song](https://people.eecs.berkeley.edu/~dawnsong/), and [Jacob Steinhardt](https://www.stat.berkeley.edu/~jsteinhardt/).\n\nDownload the [**APPS dataset here**](https://people.eecs.berkeley.edu/~hendrycks/APPS.tar.gz). (~1.3GB)\n\nThis repository contains both training and evaluation code.\n\nFine-tuned GPT-2 1.5B and GPT-Neo 2.7B weights are available [here](https://drive.google.com/file/d/1XW1Od9L-5l9zXl1HUCyER5pS9zQTbIvU/view?usp=sharing).\n\nFor other benchmarks of enormous Transformers, see a dataset which tests ability in [competition math](https://github.com/hendrycks/math), a dataset which tests knowledge of [ethics](https://github.com/hendrycks/ethics), and [a dataset spanning 50+ academic subjects](https://github.com/hendrycks/test).\n\n## How to Use\n\nThe training instructions are specified in [train/README](train/README.md) and similarly the evaluation instructions are specified in [eval/README](eval/README.md).\n\n### Hugging Face\n\nThe dataset is also available in [Hugging Face datasets](https://huggingface.co/datasets/codeparrot/apps) under apps.\n\n## Citation\n\nIf you find this useful in your research, please consider citing\n\n    @article{hendrycksapps2021,\n      title={Measuring Coding Challenge Competence With APPS},\n      author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt},\n      journal={NeurIPS},\n      year={2021}\n    }","github_created_at":"2021-04-24T20:43:37+00:00","created_at":"2026-07-11T23:46:40.753703+00:00","updated_at":"2026-10-08T06:00:41.500483+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"code-generation","name":"code-generation"},{"slug":"program-synthesis","name":"program-synthesis"}],"trust":{"provenance":{"is_fork":false,"github_id":361262700,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-08T06:00:40.723Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":840,"last_release_at":null,"stars_delta_30d":2,"open_issues_delta_30d":2},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":77,"high_count":0,"last_scan_at":"2026-07-11T23:46:43.102Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-10-08T06:00:41.182Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-10-08T06:00:41.182Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-10-08T06:00:41.182Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need benchmarking datasets specifically tailored for assessing the performance of your AI in solving programming tasks","For training and evaluating the efficacy of large language models like GPT-2 or GPT-Neo on code generation"],"when_not_to_use":["If you solely require general datasets without a focus on coding challenges","When your use case does not involve using Python-based tools for developing machine learning applications that include program synthesis and code generation"],"source":"enrich:decision_facts","observed_at":"2026-07-17T05:16:10.726Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"APPS offers a benchmark to evaluate the competence of large language models on coding challenges using its datasets."}]}}