{"data":{"slug":"facebookresearch-coder-reviewer-reranking","name":"coder_reviewer_reranking","tagline":"Official code for paper on Coder Reviewer Reranking in Code Generation","github_url":"https://github.com/facebookresearch/coder_reviewer_reranking","owner":"facebookresearch","repo":"coder_reviewer_reranking","owner_avatar_url":"https://avatars.githubusercontent.com/u/16943930?v=4","primary_language":"Python","stars":45,"forks":9,"topics":[],"archived":true,"github_pushed_at":"2023-02-14T19:22:12+00:00","maintenance_label":"Archived","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/facebookresearch-coder-reviewer-reranking","markdown_url":"https://www.graphcanon.com/tools/facebookresearch-coder-reviewer-reranking.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/facebookresearch-coder-reviewer-reranking","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=facebookresearch-coder-reviewer-reranking","description":"Official code release for the paper Coder Reviewer Reranking for Code Generation.","homepage_url":null,"license":"Other","open_issues":1,"watchers":1,"ai_summary":"Provides tools and data sets to rerank generated codes based on specific criteria such as humaneval, MBPP, Spider, and NL2BASH.","readme_excerpt":"# Coder Reviewer Reranking for Code Generation\n\n\n\nOfficial code release for the paper [Coder Reviewer Reranking for Code Generation](https://arxiv.org/abs/2211.16490).\n\n\n## Setup\n### Downloading data and cached outputs\n1. For convenience, we include data used for this project in [`dataset.zip`](https://dl.fbaipublicfiles.com/coder-reviewer/dataset.zip). You need to download and unzip this file before using this repo.\nThese include \n- [HumanEval](https://github.com/openai/human-eval). We also include the prompt used in the [CodeT](https://github.com/microsoft/CodeT/tree/main/CodeT) paper\n- [MBPP](https://github.com/google-research/google-research/tree/master/mbpp), which includes both the sanitized version and the initial version.\n- [Spider](https://github.com/taoyds/spider) includes the evaluation script and the data. We also include the cached outputs from executing the groundtruth SQL queries.\n- [NL2BASH](https://github.com/TellinaTool/nl2bash/tree/master/data)\n2. Samples and precomputed execution results can be found in [`samples.zip`](https://dl.fbaipublicfiles.com/coder-reviewer/samples.zip)\n\n### Installing software environment\n1. All experiments are run with `python==3.8.13`. \n2. Install [pyminifier](https://github.com/liftoff/pyminifier/tree/master) from source.\nInstalling `pyminifier` requires reverting setup tools to an older version (`pip install setuptools==57.5.0`). \nFor other issues of installing `pyminifier`, checkout their [issues](https://github.com/liftoff/pyminifier/issues) for potential fixes.\n3. Install `torch==1.12.1`. You should install a distribution that matches your hardware environment \n4. Install the other packages by \n```bash\npip install -r requirements.txt\n```\n\n## Usage\n\n### Running the selector with released outputs\n1. We release samples obtained from the OpenAI codex API in `samples.zip`. Unzipping this file, you should see a folder with the below structure\n```bash\nsamples\n├── codex-cushman\n│   ├── codet_humaneval\n│   └── mbpp_sanitized\n├── codex001\n└── codex002\n```\nWe will go over the code/commands you need to collect these samples in a later section.\n2. Run the following script to compare different reranking methods.\n```bash\nmodel=\"codex002\"\ndataset=\"mbpp_sanitized\"\noutdir=\"result_db\"\npython sample_selectors.py --model ${model} \\\n    --num_samples_end 25 \\\n    --num_samples_gap 5 \\\n    --data_path samples \\\n    --out_dir ${outdir} \\\n    --dataset ${dataset} \\\n    --num_procs 10 \\\n    --num_bootstraps 50 \\\n    --temperature 0.4 \\\n    --verbose\\\n```\n3. We have included the execution results of all generated samples in the `samples.zip`. If you want to execute the generated programs yourself, you can run the following command. Typically, we leverage aggressive multiprocessing to speed up this process. You can change the number of processes by modifying `nprocs`.\nModify the `model` and `dataset` arguments to execute other models and datasets. \n```bash\nmodel=\"codex002\"\ndataset=\"codet_humaneval\"\nnprocs=25\ntorchrun --nproc_per_node=${nprocs} multi_exec.py --temperature 0.4 --world_size 25 --dataset ${dataset} --in_data_path samples/${model} --batch_size 4 --num_seeds 25 --num_samples 5 --num_prompts 0\n```\n\nThe outputs will look like and a dictionary object containing the result will be saved into `result_db`\n```\nsum_logprob 0.5587 0.01\navg_logprob 0.5832 0.01\navg_reverse_logprob 0.5626 0.01\nrandom 0.5562 0.01\nsumreverselogprob-ensemble#0.5 0.6152 0.01\navgreverselogprob-ensemble#0.5 0.5963 0.01\nexecutability-sum_logprob 0.5976 0.01\nexecutability-avg_logprob 0.6049 0.01\nexecutability-avg_reverse_logprob 0.5952 0.01\nexecutability-random 0.5881 0.01\nexecutability-sumreverselogprob-ensemble#0.5 0.6440 0.01\nexecutability-avgreverselogprob-ensemble#0.5 0.6159 0.01\nmbr_exec 0.6389 0.01\noracle 0.7891 0.01\n```\n\n### Collecting Samples\n1. the below example command collects 125 (5x25) samples for zeroshot humaneval with codex002. explore `collect*.py` for collecting samples on other datasets. These scripts collect","github_created_at":"2022-11-29T02:28:37+00:00","created_at":"2026-07-11T23:45:14.731074+00:00","updated_at":"2026-10-07T06:00:49.090442+00:00","categories":[{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"}],"tags":[{"slug":"code-generation","name":"code-generation"},{"slug":"data-sets","name":"data sets"},{"slug":"evaluation","name":"evaluation"},{"slug":"reranking","name":"reranking"}],"trust":{"provenance":{"is_fork":false,"github_id":571849314,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-07T06:00:48.319Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1330,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":85,"high_count":0,"last_scan_at":"2026-07-11T23:45:16.853Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-10-07T06:00:48.775Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-10-07T06:00:48.775Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-10-07T06:00:48.775Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Specifically relies on installing pyminifier which might necessitate reverting setuptools to an older version (57.5.0).","Involves the use of torch==1.12.1, requiring a distribution compatible with your hardware."],"min_ram_gb":4,"requires_docker":false},"constraints":{"min_ram_gb":4,"requires_docker":false},"when_to_use":["When you need to refine and enhance the output from initial code generation models, such as the OpenAI codex API.","In cases where specific benchmarks like HumanEval, MBPP, or custom datasets are crucial for evaluating generated code quality.","If you are working with Python version 3.8.13 and require integration of pyminifier for minification tasks."],"when_not_to_use":["When using a different Python version than 3.8.13 as it might result in compatibility issues.","If real-time code generation is required, given the setup intricacies and specific dependencies that may affect quick turnaround times."],"source":"enrich:decision_facts","observed_at":"2026-07-17T05:19:39.324Z"},"constraint_facets":{"min_ram_gb":4,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Min 4 GB RAM; Specifically relies on installing pyminifier which might necessitate reverting setuptools to an older version (57.5.0).; Involves the use of torch==1.12.1, requiring a distribution compatible with your hardware."},{"label":"Adopt for","value":"coder_reviewer_reranking is specialized for reranking generated code snippets based on predefined criteria to improve their quality and reliability."},{"label":"License detail","value":"Other license"}]}}