prometheus-eval
prometheus-eval/prometheus-eval
Evaluate your LLM's response with Prometheus and GPT4
Overview
Prometheus-Eval is a Python-based tool for evaluating the performance of large language models using metrics from Prometheus and interactions with GPT-4.
Categories
Tags
Similar tools
ollama
ollama/ollama
Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
prompts.chat
f/prompts.chat
The world's largest open-source prompt library for AI
transformers
huggingface/transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models
langflow
langflow-ai/langflow
Langflow is a powerful platform for building and deploying AI-powered agents and workflows.
open-webui
open-webui/open-webui
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
awesome-llm-apps
Shubhamsaboo/awesome-llm-apps
100+ AI Agent & RAG apps you can actually run — clone, customize, ship.
Install
pip install prometheus-evalREADME
🔥 Prometheus-Eval 🔥
⚡ A repository for evaluating LLMs in generation tasks 🚀 ⚡
Latest News 🔥
-
[2025/04] We release the latest iteration of Prometheus: M-Promethues (3B, 7B, & 14B)!
- They outperform previous open LLM judges on multilingual meta-evaluation benchmarks (MM-Eval and M-RewardBench), and achieves exceptional results on literary translation evaluation.
- The models also perform strongly in English, with the 7B and 14B models surpassing Prometheus 2 7B and 8x7B on RewardBench, respectively.
- When used as judges at inference time, they significantly boost multilingual generation quality.
- Checkout our paper, where we present extensive ablations to uncover the key factors behind effective multilingual judge training.
-
[2024/06] We release the BiGGen-Bench and Prometheus 2 BGB (8x7B)!
- BiGGen-Bench features 9 core capabilities, 77 tasks, and 765 meticulously crafted instances, each with specific evaluation criteria.
- We evaluated 103 frontier language models by 5 state-of-the-art evaluator language models and analyzed the findings in our paper.
- We continually trained Prometheus 2 8x7B on BiGGen-Bench evaluation trace and built our most capable evaluator LM Prometheus 2 BGB, even surpassing Claude-3-Opus on absolute grading tasks.
- Checkout our dataset, evaluation results, leaderboard, interactive report, and the code!
-
[2024/05] We release Prometheus 2 (7B & 8x7B) models!
- Prometheus 2 (8x7B) is an open-source state-of-the-art evaluator language model!
- Compared to Prometheus 1 (13B), Prometheus 2 (8x7B) shows improved evaluation performances & supports assessing in pairwise ranking (relative grading) formats as well!
- It achieves a Pearson correlation of 0.6 to 0.7 with GPT-4-1106 on a 5-point Likert scale across multiple direct assessment benchmarks, including VicunaBench, MT-Bench, and FLASK.
- It also scores a 72% to 85% agreement with human judgments across multiple pairwise ranking benchmarks, including HHH Alignment, [MT Bench Human Judgment](
- Prometheus 2 (8x7B) is an open-source state-of-the-art evaluator language model!