{"data":{"slug":"huggingface-lighteval","name":"lighteval","tagline":"All-in-one toolkit for evaluating LLMs across multiple backends","github_url":"https://github.com/huggingface/lighteval","owner":"huggingface","repo":"lighteval","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":2472,"forks":506,"topics":["evaluation","evaluation-framework","evaluation-metrics","huggingface"],"archived":false,"github_pushed_at":"2026-06-29T13:03:33+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/huggingface-lighteval","markdown_url":"https://www.graphcanon.com/tools/huggingface-lighteval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-lighteval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-lighteval","description":"Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends","homepage_url":"https://huggingface.co/docs/lighteval/en/index","license":"MIT","open_issues":347,"watchers":26,"ai_summary":"Lighteval provides a comprehensive set of tools and frameworks to evaluate the performance of language models (LLMs) on different computing platforms or backend infrastructures.","readme_excerpt":"## ⚡️ Installation\n\n> **Note**: lighteval is currently *completely untested on Windows*, and we don't support it yet. (*Should be fully functional on Mac/Linux*)\n\n```bash\npip install lighteval\n```\n\nLighteval allows for *many extras* when installing, see [here](https://huggingface.co/docs/lighteval/installation) for a **complete list**.\n\nIf you want to push results to the **Hugging Face Hub**, add your access token as\nan environment variable:\n\n```shell\nhf auth login\n```","github_created_at":"2024-01-26T13:15:39+00:00","created_at":"2026-07-11T10:34:37.952113+00:00","updated_at":"2026-07-12T06:53:49.450695+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"evaluation","name":"evaluation"},{"slug":"evaluation-framework","name":"evaluation framework"},{"slug":"evaluation-metrics","name":"evaluation-metrics"},{"slug":"huggingface","name":"huggingface"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":748650671,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:34:38.577Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":11,"last_release_at":"2025-11-24T15:57:33Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:39.455Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T04:34:59.238Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-12T04:34:59.238Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-12T04:34:59.238Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-12T04:34:59.238Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to evaluate the performance of various LLMs on different backend infrastructures, especially if you are working within Mac/Linux environments.","If your evaluation process involves pushing results to the Hugging Face Hub and integrating with its ecosystem."],"when_not_to_use":["Avoid Lighteval for evaluations on Windows systems as it is currently untested and not supported there.","Should you require a solution that does not integrate with or depend on the Hugging Face ecosystem, Lighteval might not fulfill your needs."],"source":"enrich:decision_facts","observed_at":"2026-07-12T04:35:20.157Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Lighteval is designed for evaluating language models across multiple backends. It integrates well with Hugging Face and provides a wide range of extras, making it particularly handy in non-Windows environments."}]}}