lm-evaluation-harness
A framework for few-shot evaluation of language models.
GraphCanon updated Oct 12, 2026 · GitHub synced Oct 12, 2026
70views this month
Decision brief
lm-evaluation-harness is a Python framework for evaluating language models in various parallelism modes using different checkpoint formats, compatible with the Megatron-LM backend.
Good fit when
- - When you need to evaluate large language models across multiple GPUs in data or tensor parallel configurations.
Avoid when
- - If your evaluation setup requires pipeline parallelism not currently supported by this framework.
Observed Jul 11, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Active (27d since push)
- As of Oct 12, 2026
- Provenance
- Not a fork · Organization account
- As of Oct 12, 2026
- Security (OSV)
- No lockfile
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install lm-evaluation-harnessSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides a method to evaluate various aspects of trained language models using different parallelism modes and checkpoint formats.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Oct 12, 2026
- Languages
- python
Source: github.language+pyproject.toml · Oct 12, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Tags
README
Install To install the package from the github repository, run: Set environment variable pointing to Megatron LM installation export MEGATRON PATH=/path/to/Megatron LM bash lm eval model megatron lm \ model args load=/path/to/checkpoint,tokenizer type=HuggingFaceTokenizer,tokeniz...
For agents
This page has a .md twin and JSON over the API.