GraphCanon updated 2w · GitHub synced 2w
Decision brief
OctoPack is an instruction tuning code large language models repository providing detailed components for model training with data retrieval.
Good fit when
- When you need to fine-tune StarCoder or CodeGeeX2 on commit message datasets formatted as instructions
- For research purposes aiming deep integration of GitHub commits in code generation tasks
Avoid when
- If your project does not require instruction tuning and focuses solely on general model improvements
- When your data source is limited to English or a few languages, excluding the need for broad linguistic coverage as provided by CommitPack
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (545d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/bigcode-project/octopackSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A repository that provides an overview of components for instruction tuning code large language models.
Capability facts
- Languages
- jupyter notebook
Source: github.language · Aug 5, 2026
Categories
Tags
README
OctoPack: Instruction Tuning Code Large Language Models
This repository provides an overview of all components from the paper OctoPack: Instruction Tuning Code Large Language Models. Link to 5-min video on the paper presented by Niklas Muennighoff.
- Overview
- Data
- CommitPack
- CommitPackFT
- Other
- Evaluation
- Run
- Creation
- Training
- OctoCoder
- OctoGeeX
- SantaCoder Finetuning
- SantaCoder Pretraining (SantaCoderPack)
- Other
- Visuals
- Licenses
- Citation
Overview
| Data | CommitPack | 4TB of GitHub commits across 350 programming languages |
|---|---|---|
| CommitPackFT | Filtered version of CommitPack for high-quality commit messages that resemble instructions | |
| Model | OctoCoder | StarCoder (16B parameters) instruction tuned on CommitPackFT + OASST |
| OctoGeeX | CodeGeeX2 (6B parameters) instruction tuned on CommitPackFT + OASST | |
| Evaluation | HumanEvalPack | Extension of OpenAI's HumanEval to cover 3 scenarios across 6 languages |
Data
CommitPack
CommitPack is uploaded here. To recreate:
- BigQuery SQL: Use BigQuery to select the commit data from the GitHub action data. All SQL commands can be found in
dataset/commitpack/sql. They are executed in order starting with the first one to to the fifth one. They are separated and executed one-by-one as BigQuery was raisingResources exceedederrors during query execution when running all in a single statement. After each SQL query a dataset is created and named as indicated in the filename. E.g. after executingsql_1_commits_table_base.sql, you would name the output datasetcommits_table_base, which is then referenced in the 2nd statement. - Export: From BigQuery export the dataset after the final SQL statement inside GCP to a bucket as parquet files.
- Upload to HF: Use a GCP compute instance to copy all the parquet files into a Hugging Face dataset and push it. The resulting dataset contains metadata on the commits, CommitPackMeta
- Scrape GitHub: Run the script at
dataset/commitpack/scrape_github.pyto download the files prior and after each git commit from GitHub. It contains some basic filters to remove noise files (relying on the extensions file atdataset/commitpack/programming_languages.json) and then uses multi-threading and multi-processing for scraping. It is recommended to run it on a very large instance. - Shard (optional): Depending on the size of your files, you may want to shard them at this point using the script at
dataset/commitpack/shard.sh - Opt-out & languages: Run the script at
dataset/commitpack/licenses_langs.pyto remove repositories from users who opted out of the step (first part with__main__, needs to be uncommented) and split the large files from the prior step into files for each programming language (second part with__main__, currently uncommented). You will likely have to change some of the path names and uncomment parts as necessary - Shard (optional): Using the script at
dataset/commitpack/shard.pyyou
For agents
This page has a .md twin and JSON over the API.