OpenCoder
</h1>
 <p align="center">⚡ The Open Cookbook for Top-Tier Code Large Language Models ⚡</p>

🏠Home Page   |   🤗Model   |   📊Dataset   |   📄Paper  ｜ 🚀Demo  

News

• 🔥🔥🔥 2024/12/08 We have released our pretraining data cleaning pipeline: opc_data_filtering. Try to use this pipeline to create your own high-quality code pretraining corpus!
• 🔥 2024/11/19 We have released intermedidate checkpoints during our pretraining stage: 🤗 OpenCoder-1.5B-Base-Checkpoints and 🤗 OpenCoder-8B-Base-Checkpoints.
• 🔥 2024/11/15 We have released meta data of RefineCode 📊 RefineCode-code-corpus-meta. You can collect your own RefineCode referring to this dataset!
• 🔥 2024/11/12 We have released our efficient CodeLLM evaluation framework: OpenCodeEval.
• 🔥 2024/11/12 We have released high-quality annealing data 📊 opc-annealing-corpus, which includes algorithmic-corpus along with corresponding synthetic data.
• 🔥 2024/11/11 We have released 55B of recalled pages from Fineweb, including 📊 fineweb-code-corpus and 📊 fineweb-math-corpus.
• 🔥 2024/11/09 We have released 4.5M Post-training data: 📊 Dataset.
• 🔥 2024/11/08 We have released our models! Please download them from 🤗 Model.
• 🔥 2024/11/07 We have released our paper on Arxiv: 📄 OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models.

Releases

• Data cleaning pipeline
• RefineCode: Code-related web data
• RefineCode: Metadata of raw code data
• Intermedidate Checkpoints
• CodeLLM evaluation framework: OpenCodeEval
• High-quality annealing data
• Post-training data
• Final model weights
• Paper

We are working hard to release all those resources! 💪

Introduction

OpenCoder is an open and reproducible code LLM family which includes 1.5B and 8B base and chat models, supporting both English and Chinese languages. Starting from scratch, OpenCoder is pretrained on 2.5 trillion tokens composed of 90% raw code and 10% code-related web data, and supervised finetuned on over 4.5M high-quality SFT examples, finally reaching the performance of top-tier code LLMs. We provide not only model weights and inference code, but also the reproducible training data, the complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols. Empowering researchers to build and innovate, OpenCoder is your open fo