About

This repository contains two components: Megatron-LM and Megatron Core.

Megatron-LM is a reference example that includes Megatron Core plus pre-configured training scripts, ideal for research teams, learning distributed training, and quick experimentation.

Megatron Core is a composable library with GPU-optimized building blocks for custom training frameworks. It provides transformer building blocks, advanced parallelism strategies (TP, PP, DP, EP, and CP), mixed precision support (FP16, BF16, FP8, and FP4), and model architectures, ideal for framework developers and ML engineers building custom training pipelines.

Megatron Bridge provides bidirectional Hugging Face ↔ Megatron checkpoint conversion with production-ready recipes.

Getting Started

Install from PyPI:

uv pip install megatron-core

Or clone and install from source:

git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
uv pip install -e .

Note: Building from source can use a lot of memory. If the build runs out of memory, limit parallel compilation jobs by setting MAX_JOBS (for example, MAX_JOBS=4 uv pip install -e .).

For NVIDIA GPU Cloud (NGC) container setup and all installation options, review the Installation Guide.

• Your First Training Run - End-to-end training examples with data preparation
• Parallelism Strategies - Scale training across GPUs with TP, PP, DP, EP, and CP
• Contribution Guide - How to contribute to Megatron Core

Latest News

• [2026/05] DeepSeek-V4 initial support - Megatron Core's dev branch includes the initial DeepSeek-V4 implementation; Megatron Bridge provides conversion, inference, and pretraining recipes.
• [2026/04] Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron - Muon and other emerging optimizers are now supported in Megatron Core via the new Emerging-Optimizers library.
• [2026/03] Scalable Training of Mixture-of-Experts Models with Megatron Core - Technical report on scaling MoE training with integrated optimizations for memory, communication, and computation.
• [2026/03] Implementing Falcon-H1 Hybrid Architecture in Megatron Core - Technology Innovation Institute (TII) contributes Falcon-H1 hybrid transformer-Mamba architecture and BitNet ternary quantization support to Megatron Core.
• [2026/03] Megatron Core Roadmap - Roadmap for upcoming Megatron Core features and improvements.
• [2026/03] Deprecating Python 3.10 support: The upcoming 0.17.0 release drops Python 3.10 support. Downstream applications must raise their lower boundary to 3.12 to stay compatible with Megatron Core.
• [2026/01] **[Dynamic Context Parallelism](https://developer.nvidia.com/blog/speeding-up-variable-length-training-with-dynamic-context-parallelism-and-nvidia-megatro