Home/Evaluation & Observability/helm

helm

stanford-crfm/helm

Holistic, reproducible and transparent evaluation of foundation models

GraphCanon updated Oct 12, 2026 · GitHub synced Oct 12, 2026

50views this month

2.9k stars421 forksLast push Sep 1, 2026 Python Apache-2.0

Decision brief

Helm is an open-source Python framework for evaluating foundation models, including LLMs and multimodal models. It emphasizes holistic, reproducible, and transparent evaluation processes.

Good fit when

When you need a comprehensive tool to evaluate the performance of large language models (LLMs) and other types of foundation models in a standardized way.
If your research or development project requires transparent and reproducible evaluation protocols, Helm's established framework supports rigorous testing methods.

Avoid when

Helm may not be suitable if you are working with smaller scale projects that do not require extensive, holistic evaluation capabilities associated with foundation models.
If your framework of choice already provides sufficient evaluation tools or processes for foundation models, adding Helm might introduce unnecessary complexity.

Observed Jul 12, 2026 · Source: enrich:decision_facts

helm alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Steady (41d since push); As of Oct 12, 2026
Provenance: Not a fork · Organization account; As of Oct 12, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install helm

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

Framework for evaluating LLMs and LLM systems with an open-source registry of benchmarks.

LLM Evaluation Framework.

lm-evaluation-harness

A framework for few-shot evaluation of language models.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

An open-source Python framework for evaluating foundation models, including LLMs and multimodal models.

Capability facts

CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Oct 12, 2026
Languages: python
Source: github.language+pyproject.toml · Oct 12, 2026

Categories

Evaluation & Observability

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Oct 12, 2026)

pip install crfm-helm

Tags

evaluation foundation models framework language models

README

Quick Start Install the package from PyPI: Run the following in your shell: ```sh

For agents

This page has a .md twin and JSON over the API.

/tools/stanford-crfm-helm.md GET /api/graphcanon/tools/stanford-crfm-helm GET /api/graphcanon/graph?tool=stanford-crfm-helm

Was this helpful?

Anonymous feedback helps us improve pages and translations.