{"data":{"slug":"eleutherai-lm-evaluation-harness","name":"lm-evaluation-harness","tagline":"A framework for few-shot evaluation of language models.","github_url":"https://github.com/EleutherAI/lm-evaluation-harness","owner":"EleutherAI","repo":"lm-evaluation-harness","owner_avatar_url":"https://avatars.githubusercontent.com/u/68924597?v=4","primary_language":"Python","stars":13253,"forks":3404,"topics":["evaluation-framework","language-model","transformer"],"archived":false,"github_pushed_at":"2026-06-24T19:51:08+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/eleutherai-lm-evaluation-harness","markdown_url":"https://www.graphcanon.com/tools/eleutherai-lm-evaluation-harness.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/eleutherai-lm-evaluation-harness","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=eleutherai-lm-evaluation-harness","description":"A framework for few-shot evaluation of language models.","homepage_url":"https://www.eleuther.ai","license":"MIT","open_issues":907,"watchers":51,"ai_summary":"Provides a method to evaluate various aspects of trained language models using different parallelism modes and checkpoint formats.","readme_excerpt":"## Install\n\nTo install the `lm-eval` package from the github repository, run:\n\n```bash\ngit clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness\ncd lm-evaluation-harness\npip install -e .\n```\n\n---\n\n# Set environment variable pointing to Megatron-LM installation\nexport MEGATRON_PATH=/path/to/Megatron-LM\n```\n\n**Basic usage (single GPU):**\n\n```bash\nlm_eval --model megatron_lm \\\n    --model_args load=/path/to/checkpoint,tokenizer_type=HuggingFaceTokenizer,tokenizer_model=/path/to/tokenizer \\\n    --tasks hellaswag \\\n    --batch_size 1\n```\n\n**Supported checkpoint formats:**\n- Standard Megatron checkpoints (`model_optim_rng.pt`)\n- Distributed checkpoints (`.distcp` format, auto-detected)\n\n#### Parallelism Modes\n\nThe Megatron-LM backend supports the following parallelism modes:\n\n| Mode | Configuration | Description |\n|------|---------------|-------------|\n| Single GPU | `devices=1` (default) | Standard single GPU evaluation |\n| Data Parallelism | `devices>1, TP=1` | Each GPU has a full model replica, data is distributed |\n| Tensor Parallelism | `TP == devices` | Model layers are split across GPUs |\n| Expert Parallelism | `EP == devices, TP=1` | For MoE models, experts are distributed across GPUs |\n\n> [!Note]\n> - Pipeline Parallelism (PP > 1) is not currently supported.\n> - Expert Parallelism (EP) cannot be combined with Tensor Parallelism (TP).\n\n**Data Parallelism (4 GPUs, each with full model replica):**\n\n```bash\ntorchrun --nproc-per-node=4 -m lm_eval --model megatron_lm \\\n    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,devices=4 \\\n    --tasks hellaswag\n```\n\n**Tensor Parallelism (TP=2):**\n\n```bash\ntorchrun --nproc-per-node=2 -m lm_eval --model megatron_lm \\\n    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,devices=2,tensor_model_parallel_size=2 \\\n    --tasks hellaswag\n```\n\n**Expert Parallelism for MoE models (EP=4):**\n\n```bash\ntorchrun --nproc-per-node=4 -m lm_eval --model megatron_lm \\\n    --model_args load=/path/to/moe_checkpoint,tokenizer_model=/path/to/tokenizer,devices=4,expert_model_parallel_size=4 \\\n    --tasks hellaswag\n```\n\n**Using extra_args for additional Megatron options:**\n\n```bash\nlm_eval --model megatron_lm \\\n    --model_args load=/path/to/checkpoint,tokenizer_model=/path/to/tokenizer,extra_args=\"--no-rope-fusion --trust-remote-code\" \\\n    --tasks hellaswag\n```\n\n> [!Note]\n> The `--use-checkpoint-args` flag is enabled by default, which loads model architecture parameters from the checkpoint. For checkpoints converted via Megatron-Bridge, this typically includes all necessary model configuration.\n\n#### Multi-GPU evaluation with OpenVINO models\n\nPipeline parallelism during evaluation is supported with OpenVINO models\n\nTo enable pipeline parallelism, set the `model_args` of `pipeline_parallel`. In addition, you also have to set up `device` to value `HETERO:<GPU index1>,<GPU index2>` for example `HETERO:GPU.1,GPU.0` For example, the command to use pipeline parallelism of 2 is:\n\n```bash\nlm_eval --model openvino \\\n    --tasks wikitext \\\n    --model_args pretrained=<path_to_ov_model>,pipeline_parallel=True \\\n    --device HETERO:GPU.1,GPU.0\n```","github_created_at":"2020-08-28T00:09:15+00:00","created_at":"2026-07-11T10:34:33.979221+00:00","updated_at":"2026-07-11T15:15:36.337356+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"data-parallelism","name":"data-parallelism"},{"slug":"evaluation-framework","name":"evaluation framework"},{"slug":"expert-parallelism","name":"expert-parallelism"},{"slug":"language-model","name":"language-model"},{"slug":"lm-evaluation-harness","name":"lm-evaluation-harness"},{"slug":"model-evaluation","name":"model-evaluation"},{"slug":"tensor-parallelism","name":"tensor-parallelism"},{"slug":"transformer","name":"transformer"}],"trust":{"provenance":{"is_fork":false,"github_id":290909192,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:34:34.559Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":1,"days_since_push":16,"last_release_at":"2026-05-11T13:04:19Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:35.382Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T15:15:23.310Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T15:15:23.310Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T15:15:23.310Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T15:15:23.310Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When you need to evaluate large language models across multiple GPUs in data or tensor parallel configurations."],"when_not_to_use":["- If your evaluation setup requires pipeline parallelism not currently supported by this framework."],"source":"enrich:decision_facts","observed_at":"2026-07-11T15:15:36.040Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"lm-evaluation-harness is a Python framework for evaluating language models in various parallelism modes using different checkpoint formats, compatible with the Megatron-LM backend."}]}}