{"data":{"slug":"theoxo-self-repair","name":"self-repair","tagline":"[ICLR 2024]: Is Self-Repair a Silver Bullet for Code Generation?","github_url":"https://github.com/theoxo/self-repair","owner":"theoxo","repo":"self-repair","owner_avatar_url":"https://avatars.githubusercontent.com/u/25453700?v=4","primary_language":"Python","stars":15,"forks":3,"topics":[],"archived":true,"github_pushed_at":"2024-05-02T13:56:55+00:00","maintenance_label":"Archived","url":"https://www.graphcanon.com/tools/theoxo-self-repair","markdown_url":"https://www.graphcanon.com/tools/theoxo-self-repair.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/theoxo-self-repair","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=theoxo-self-repair","description":"[ICLR 2024]: Is Self-Repair a Silver Bullet for Code Generation?","homepage_url":null,"license":null,"open_issues":1,"watchers":1,"ai_summary":null,"readme_excerpt":"# [ICLR 2024]: Is Self-Repair a Silver Bullet for Code Generation?\n\nThis is is the accompanying repository for the paper [Is Self-Repair a Silver Bullet for Code Generation?](https://openreview.net/forum?id=y0GJXRungR), presented at the [Twelfth International Conference on Learning Representations](https://iclr.cc/Conferences/2024) (Vienna, May 2024).\nIt contains source code used to run the experiments; the resulting data; as well as scripts to replicate the data analysis and figures from the paper.\n\nTo install the libraries needed to run the code and analysis scripts, you can use `pip install -r requirements.txt`.\n\n## TL;DR: Replicating the Figures\n\nAll figures in the paper can be replicated by running `cd paper && make figures`. This will use pre-computed results of the data analysis, and will place the figures in `paper/figures/`.\nIf you instead want to do all of the data analysis from scratch, run `APPS_DIR=<path to my APPS directory> cd paper && make all`; note that this requires having APPS installed locally.\n\nN.B.: This repository does not contain the data collected during the human study, due to IRB policies.\n\n## Bibtex Citation\n```\n@inproceedings{olausson2024repair,\n\ttitle        = {Is Self-Repair a Silver Bullet for Code Generation?},\n\tauthor       = {Theo X. Olausson and Jeevana Priya Inala and Chenglong Wang and Jianfeng Gao and Armando Solar-Lezama},\n\tyear         = 2024,\n\tbooktitle    = {International Conference on Learning Representations (ICLR)}\n}\n```\n\n## A Note on HumanEval \n\n*Note: the below only applies if you want to use this code base to run new self-repair experiments on HumanEval yourself. You do not need to worry about this if you are merely interested in replicating the figures and results from this paper.*\n\nThis code base uses a modified version of HumanEval, in which it is easier to extract error messages from failed assertions. This can be downloaded from [people.csail.mit.edu/theoxo/data/HumanEval_with_assertion_messages.jsonl.gz.gpg](https://people.csail.mit.edu/theoxo/data/HumanEval_with_assertion_messages.jsonl.gz.gpg); you can then decrypt it with `gpg -d` using the password `theoxoiclr2024` and unpack it with `gunzip`, after which it can be used as a drop-in replacement for `HumanEval.jsonl` in your local installation of HumanEval.\n\n## A Note on APPS\n\n*Note: the below only applies if you want to use this code base to run new self-repair experiments on APPS yourself. You do not need to worry about this if you are merely interested in replicating the figures and results from this paper.*\n\nDue to dependencies on an internal project, one function (`exec_sample`) has been left unimplemented in `src/apps/apps.py`. If you want to make use of the APPS part of the source code, you must implement this function; see the doc-string for pointers.\n\n## Repository Structure\n- `src/`: source code used to run the experiments.\n    - `apps/`: source code for experiments on APPS.\n    - `humaneval/`: source code for experiments on humaneval.\n- `paper/`: data and scripts used to analyze and plot the results of the experiments.\n    - `Makefile`: makefile to reproduce figures (`make figures`), run the analysis scripts (`make analysis`) or both (`make all`)\n    - `analysis/sample-and-estimate.py`: Python script to generate bootstrapped estimates of pass rates at various budgets.\n    - `data/`:\n        - `calculate-token-counts.py`: Python script to add counts for how many tokens were used to generate the programs/feedback/repairs. Used for pass@t metrics in Appendix A.\n        - `apps/`: data from APPS experiments, with bash scripts to analyze the data and plot the results.\n        - `humaneval/`: data from humaneval experiments, with bash scripts to analyze the data and plot the results.\n    - `plotting/`: Python scripts to generate the types of figures used in the paper.\n\n## Data Format\nThe data generated by the models can be found by de-compressing the tarballs `paper/data/apps/apps-data.tar.bz2` and `paper/data/hum","github_created_at":"2024-01-26T12:10:30+00:00","created_at":"2026-07-11T23:45:01.395813+00:00","updated_at":"2026-07-11T23:45:09.535361+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":748627504,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:45:06.929Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":800,"last_release_at":null},"security_summary":{"status":"ok","scanner":"osv@v1","low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:45:07.513Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:45:06.646Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:45:06.646Z"}}}}