{"data":{"slug":"bigcode-project-octopack","name":"octopack","tagline":"🐙 OctoPack: Instruction Tuning Code Large Language Models","github_url":"https://github.com/bigcode-project/octopack","owner":"bigcode-project","repo":"octopack","owner_avatar_url":"https://avatars.githubusercontent.com/u/110470554?v=4","primary_language":"Jupyter Notebook","stars":479,"forks":29,"topics":[],"archived":false,"github_pushed_at":"2025-02-05T19:27:52+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/bigcode-project-octopack","markdown_url":"https://www.graphcanon.com/tools/bigcode-project-octopack.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bigcode-project-octopack","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bigcode-project-octopack","description":"🐙 OctoPack: Instruction Tuning Code Large Language Models","homepage_url":"https://arxiv.org/abs/2308.07124","license":"MIT","open_issues":14,"watchers":9,"ai_summary":null,"readme_excerpt":"# OctoPack: Instruction Tuning Code Large Language Models\n\n\n\nThis repository provides an overview of all components from the paper [OctoPack: Instruction Tuning Code Large Language Models](https://arxiv.org/abs/2308.07124). [Link to 5-min video](https://www.youtube.com/watch?v=vtsQ294gkJ4&t=13s&ab_channel=NiklasMuennighoff) on the paper presented by Niklas Muennighoff.\n\n\n\n- [Overview](#overview)\n- [Data](#data)\n - [CommitPack](#commitpack)\n - [CommitPackFT](#commitpackft)\n - [Other](#other)\n- [Evaluation](#evaluation)\n - [Run](#run)\n - [Creation](#creation)\n- [Training](#training)\n - [OctoCoder](#octocoder)\n - [OctoGeeX](#octogeex)\n - [SantaCoder Finetuning](#santacoder-finetuning)\n - [SantaCoder Pretraining (SantaCoderPack)](#santacoder-pretraining-santacoderpack)\n - [Other](#other-1)\n- [Visuals](#visuals)\n- [Licenses](#licenses)\n- [Citation](#citation)\n\n\n\n## Overview\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Data \nCommitPack4TB of GitHub commits across 350 programming languages
\nCommitPackFTFiltered version of CommitPack for high-quality commit messages that resemble instructions
Model \nOctoCoderStarCoder (16B parameters) instruction tuned on CommitPackFT + OASST
\nOctoGeeXCodeGeeX2 (6B parameters) instruction tuned on CommitPackFT + OASST
Evaluation \nHumanEvalPackExtension of OpenAI's HumanEval to cover 3 scenarios across 6 languages
\n\n## Data\n\n### CommitPack\n\nCommitPack is uploaded [here](https://huggingface.co/datasets/bigcode/commitpack). To recreate:\n\n1. **BigQuery SQL:** Use BigQuery to select the commit data from the GitHub action data. All SQL commands can be found in `dataset/commitpack/sql`. They are executed in order starting with the first one to to the fifth one. They are separated and executed one-by-one as BigQuery was raising `Resources exceeded` errors during query execution when running all in a single statement. After each SQL query a dataset is created and named as indicated in the filename. E.g. after executing `sql_1_commits_table_base.sql`, you would name the output dataset `commits_table_base`, which is then referenced in the 2nd statement.\n2. **Export:** From BigQuery export the dataset after the final SQL statement inside GCP to a bucket as parquet files.\n3. **Upload to HF:** Use a GCP compute instance to copy all the parquet files into a Hugging Face dataset and push it. The resulting dataset contains metadata on the commits, [CommitPackMeta](https://huggingface.co/datasets/bigcode/commitpackmeta)\n4. **Scrape GitHub:** Run the script at `dataset/commitpack/scrape_github.py` to download the files prior and after each git commit from GitHub. It contains some basic filters to remove noise files (relying on the extensions file at `dataset/commitpack/programming_languages.json`) and then uses multi-threading and multi-processing for scraping. It is recommended to run it on a very large instance.\n5. **Shard (optional):** Depending on the size of your files, you may want to shard them at this point using the script at `dataset/commitpack/shard.sh`\n6. **Opt-out & languages:** Run the script at `dataset/commitpack/licenses_langs.py` to remove repositories from users who opted out of the step (first part with `__main__`, needs to be uncommented) and split the large files from the prior step into files for each programming language (second part with `__main__`, currently uncommented). You will likely have to change some of the path names and uncomment parts as necessary\n7. **Shard (optional):** Using the script at `dataset/commitpack/shard.py` you","github_created_at":"2023-03-25T15:26:23+00:00","created_at":"2026-07-11T23:44:13.490018+00:00","updated_at":"2026-07-11T23:44:23.65832+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"jupyter-notebook","name":"jupyter notebook"}],"trust":{"provenance":{"is_fork":false,"github_id":618866368,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:44:14.985Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":521,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:44:15.341Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:44:14.745Z"},"languages":{"value":["jupyter notebook"],"source":"github.language","observed_at":"2026-07-11T23:44:14.745Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:44:14.745Z"}}}}