--- title: "prometheus-eval" type: "tool" slug: "prometheus-eval-prometheus-eval" canonical_url: "https://www.graphcanon.com/tools/prometheus-eval-prometheus-eval" github_url: "https://github.com/prometheus-eval/prometheus-eval" homepage_url: null stars: 1101 forks: 71 primary_language: "Python" license: "Apache-2.0" categories: ["evaluation-observability", "llm-frameworks"] tags: ["gpt4", "evaluation", "llm", "python", "litellm"] updated_at: "2026-07-07T18:47:55.422737+00:00" --- # prometheus-eval > Evaluate your LLM's response with Prometheus and GPT4 Prometheus-Eval is a Python-based tool for evaluating the performance of large language models using metrics from Prometheus and interactions with GPT-4. ## Facts - Repository: https://github.com/prometheus-eval/prometheus-eval - Stars: 1,101 · Forks: 71 · Open issues: 14 · Watchers: 2 - Primary language: Python - License: Apache-2.0 - Last pushed: 2025-04-25T03:58:37+00:00 ## Categories - [Evaluation & Observability](/categories/evaluation-observability.md) - [LLM Frameworks](/categories/llm-frameworks.md) ## Tags gpt4, evaluation, llm, python, litellm ## Related tools - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659) - [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019) - [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347) - [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (★ 151,298) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575) - [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702) - [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711) - [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,610) ## README (excerpt) ```text

Prometheus-Logo

🔥 Prometheus-Eval 🔥

arXiv Hugging Face Organization License PyPI version

⚡ A repository for evaluating LLMs in generation tasks 🚀 ⚡

**Latest News** 🔥 - [2025/04] We release the latest iteration of Prometheus: **[M-Promethues (3B, 7B, & 14B)](https://huggingface.co/collections/Unbabel/m-prometheus-67f3b17e6409b2550b698822)!** - They outperform previous open LLM judges on multilingual meta-evaluation benchmarks ([MM-Eval](https://arxiv.org/abs/2410.17578) and [M-RewardBench](https://arxiv.org/abs/2410.15522)), and achieves exceptional results on [literary translation evaluation](https://arxiv.org/abs/2410.18697). - The models also perform strongly in English, with the 7B and 14B models surpassing Prometheus 2 7B and 8x7B on [RewardBench](https://arxiv.org/abs/2403.13787), respectively. - When used as judges at inference time, they significantly boost multilingual generation quality. - Checkout our [paper](https://arxiv.org/abs/2504.04953), where we present extensive ablations to uncover the key factors behind effective multilingual judge training. - [2024/06] We release the **BiGGen-Bench** and **Prometheus 2 BGB (8x7B)**! - BiGGen-Bench features 9 core capabilities, 77 tasks, and 765 meticulously crafted instances, each with specific evaluation criteria. - We evaluated 103 frontier language models by 5 state-of-the-art evaluator language models and analyzed the findings in our [paper](https://arxiv.org/abs/2406.05761). - We continually trained Prometheus 2 8x7B on BiGGen-Bench evaluation trace and built our most capable evaluator LM [Prometheus 2 BGB](https://huggingface.co/prometheus-eval/prometheus-bgb-8x7b-v2.0), even surpassing Claude-3-Opus on absolute grading tasks. - Checkout our [dataset](https://huggingface.co/datasets/prometheus-eval/BiGGen-Bench), [evaluation results](https://huggingface.co/datasets/prometheus-eval/BiGGen-Bench-Results), [leaderboard](https://huggingface.co/spaces/prometheus-eval/BiGGen-Bench-Leaderboard), [interactive report](https://hub.zenoml.com/project/c84cfca5-71c9-4f89-aa0e-218c65c821e4/BiGGen\%20Bench\%20Results), and the [code](https://github.com/prometheus-eval/prometheus-eval/tree/main/BiGGen-Bench)! - [2024/05] We release Prometheus 2 (7B & 8x7B) models! - **Prometheus 2 (8x7B)** is an open-source state-of-the-art evaluator language model! - Compared to Prometheus 1 (13B), Prometheus 2 (8x7B) shows improved evaluation performances & supports assessing in pairwise ranking (relative grading) formats as well! - It achieves a Pearson correlation of 0.6 to 0.7 with GPT-4-1106 on a 5-point Likert scale across multiple direct assessment benchmarks, including [VicunaBench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge/data/vicuna_bench), [MT-Bench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge/data/mt_bench), and [FLASK](https://github.com/kaistAI/FLASK). - It also scores a 72% to 85% agreement with human judgments across multiple pairwise ranking benchmarks, including [HHH Alignment](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/hhh_alignment), [MT Bench Human Judgment]( ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/prometheus-eval-prometheus-eval`](/api/graphcanon/tools/prometheus-eval-prometheus-eval) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_