Install

To install the lm-eval package from the github repository, run:

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

---

Set environment variable pointing to Megatron-LM installation

export MEGATRON_PATH=/path/to/Megatron-LM

**Basic usage (single GPU):**

```bash
lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_type=HuggingFaceTokenizer,tokenizer_model=/path/to/tokenizer \
    --tasks hellaswag \
    --batch_size 1

Supported checkpoint formats:

• Standard Megatron checkpoints (model_optim_rng.pt)
• Distributed checkpoints (.distcp format, auto-detected)

Parallelism Modes

The Megatron-LM backend supports the following parallelism modes:

Mode	Configuration	Description
Single GPU	devices=1 (default)	Standard single GPU evaluation
Data Parallelism	devices>1, TP=1	Each GPU has a full model replica, data is distributed
Tensor Parallelism	TP == devices	Model layers are split across GPUs
Expert Parallelism	EP == devices, TP=1	For MoE models, experts are distributed across GPUs

[!Note]

• Pipeline Parallelism (PP > 1) is not currently supported.
• Expert Parallelism (EP) cannot be combined with Tensor Parallelism (TP).

Data Parallelism (4 GPUs, each with full model replica):

torchrun --nproc-per-node=4 -m lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,devices=4 \
    --tasks hellaswag

Tensor Parallelism (TP=2):

torchrun --nproc-per-node=2 -m lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,devices=2,tensor_model_parallel_size=2 \
    --tasks hellaswag

Expert Parallelism for MoE models (EP=4):

torchrun --nproc-per-node=4 -m lm_eval --model megatron_lm \
    --model_args load=/path/to/moe_checkpoint,tokenizer_model=/path/to/tokenizer,devices=4,expert_model_parallel_size=4 \
    --tasks hellaswag

Using extra_args for additional Megatron options:

lm_eval --model megatron_lm \
    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,extra_args="--no-rope-fusion --trust-remote-code" \
    --tasks hellaswag

[!Note] The --use-checkpoint-args flag is enabled by default, which loads model architecture parameters from the checkpoint. For checkpoints converted via Megatron-Bridge, this typically includes all necessary model configuration.

Multi-GPU evaluation with OpenVINO models

Pipeline parallelism during evaluation is supported with OpenVINO models

To enable pipeline parallelism, set the model_args of pipeline_parallel. In addition, you also have to set up device to value HETERO:<GPU index1>,<GPU index2> for example HETERO:GPU.1,GPU.0 For example, the command to use pipeline parallelism of 2 is:

lm_eval --model openvino \
    --tasks wikitext \
    --model_args pretrained=<path_to_ov_model>,pipeline_parallel=True \
    --device HETERO:GPU.1,GPU.0