OctoPack: Instruction Tuning Code Large Language Models

This repository provides an overview of all components from the paper OctoPack: Instruction Tuning Code Large Language Models. Link to 5-min video on the paper presented by Niklas Muennighoff.

• Overview
• Data
  • CommitPack
  • CommitPackFT
  • Other
• Evaluation
  • Run
  • Creation
• Training
  • OctoCoder
  • OctoGeeX
  • SantaCoder Finetuning
  • SantaCoder Pretraining (SantaCoderPack)
  • Other
• Visuals
• Licenses
• Citation

Overview

Data

CommitPack

CommitPack is uploaded here. To recreate:

1. BigQuery SQL: Use BigQuery to select the commit data from the GitHub action data. All SQL commands can be found in dataset/commitpack/sql. They are executed in order starting with the first one to to the fifth one. They are separated and executed one-by-one as BigQuery was raising Resources exceeded errors during query execution when running all in a single statement. After each SQL query a dataset is created and named as indicated in the filename. E.g. after executing sql_1_commits_table_base.sql, you would name the output dataset commits_table_base, which is then referenced in the 2nd statement.
2. Export: From BigQuery export the dataset after the final SQL statement inside GCP to a bucket as parquet files.
3. Upload to HF: Use a GCP compute instance to copy all the parquet files into a Hugging Face dataset and push it. The resulting dataset contains metadata on the commits, CommitPackMeta
4. Scrape GitHub: Run the script at dataset/commitpack/scrape_github.py to download the files prior and after each git commit from GitHub. It contains some basic filters to remove noise files (relying on the extensions file at dataset/commitpack/programming_languages.json) and then uses multi-threading and multi-processing for scraping. It is recommended to run it on a very large instance.
5. Shard (optional): Depending on the size of your files, you may want to shard them at this point using the script at dataset/commitpack/shard.sh
6. Opt-out & languages: Run the script at dataset/commitpack/licenses_langs.py to remove repositories from users who opted out of the step (first part with __main__, needs to be uncommented) and split the large files from the prior step into files for each programming language (second part with __main__, currently uncommented). You will likely have to change some of the path names and uncomment parts as necessary
7. Shard (optional): Using the script at dataset/commitpack/shard.py you