---
title: "continuous-eval"
type: "tool"
slug: "relari-ai-continuous-eval"
canonical_url: "https://www.graphcanon.com/tools/relari-ai-continuous-eval"
github_url: "https://github.com/relari-ai/continuous-eval"
homepage_url: "https://continuous-eval.docs.relari.ai/"
stars: 516
forks: 38
primary_language: "Python"
license: "Apache-2.0"
categories: ["evaluation-observability"]
tags: ["llmops", "information-retrieval", "retrieval-augmented-generation", "evaluation-metrics", "evaluation-framework", "llm-evaluation"]
updated_at: "2026-07-07T18:49:52.867716+00:00"
---
# continuous-eval
> Data-Driven Evaluation for LLM-Powered Applications
An open-source Python package designed to evaluate large language model (LLM) applications through modularized and comprehensive metric libraries, including probabilistic evaluation.
## Facts
- Repository: https://github.com/relari-ai/continuous-eval
- Homepage: https://continuous-eval.docs.relari.ai/
- Stars: 516 · Forks: 38 · Open issues: 12 · Watchers: 5
- Primary language: Python
- License: Apache-2.0
- Last pushed: 2025-01-22T23:29:16+00:00
## Categories
- [Evaluation & Observability](/categories/evaluation-observability.md)
## Tags
llmops, information-retrieval, retrieval-augmented-generation, evaluation-metrics, evaluation-framework, llm-evaluation
## Related tools
- [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (★ 151,298)
- [graphrag](/tools/microsoft-graphrag.md) - A modular graph-based Retrieval-Augmented Generation (RAG) system (★ 34,239)
- [langfuse](/tools/langfuse-langfuse.md) - Langfuse: Open source AI engineering platform for LLM evaluation, observability, and prompt management. (★ 30,626)
- [RAG_Techniques](/tools/nirdiamant-rag-techniques.md) - Advanced RAG Techniques (★ 28,398)
- [mlflow](/tools/mlflow-mlflow.md) - The open source AI engineering platform for agents, LLMs, and ML models. (★ 26,921)
- [llm-action](/tools/liguodongiot-llm-action.md) - 分享大模型技术原理和实战经验，涵盖工程化、应用落地 (★ 24,670)
- [awesome-n8n-templates](/tools/enescingoz-awesome-n8n-templates.md) - The largest open-source collection of n8n automation templates on GitHub. (★ 23,714)
- [promptfoo](/tools/promptfoo-promptfoo.md) - Test your prompts, agents, and RAGs. Evaluate LLM performance with red teaming/pentesting/vulnerability scanning. (★ 23,003)
## README (excerpt)
```text
Data-Driven Evaluation for LLM-Powered Applications
## Overview
`continuous-eval` is an open-source package created for data-driven evaluation of LLM-powered application.
## How is continuous-eval different?
- **Modularized Evaluation**: Measure each module in the pipeline with tailored metrics.
- **Comprehensive Metric Library**: Covers Retrieval-Augmented Generation (RAG), Code Generation, Agent Tool Use, Classification and a variety of other LLM use cases. Mix and match Deterministic, Semantic and LLM-based metrics.
- **Probabilistic Evaluation**: Evaluate your pipeline with probabilistic metrics
## Getting Started
This code is provided as a PyPi package. To install it, run the following command:
```bash
python3 -m pip install continuous-eval
```
if you want to install from source:
```bash
git clone https://github.com/relari-ai/continuous-eval.git && cd continuous-eval
poetry install --all-extras
```
To run LLM-based metrics, the code requires at least one of the LLM API keys in `.env`. Take a look at the example env file `.env.example`.
## Run a single metric
Here's how you run a single metric on a datum.
Check all available metrics here: [link](https://continuous-eval.docs.relari.ai/)
```python
from continuous_eval.metrics.retrieval import PrecisionRecallF1
datum = {
"question": "What is the capital of France?",
"retrieved_context": [
"Paris is the capital of France and its largest city.",
"Lyon is a major city in France.",
],
"ground_truth_context": ["Paris is the capital of France."],
"answer": "Paris",
"ground_truths": ["Paris"],
}
metric = PrecisionRecallF1()
print(metric(**datum))
```
## Run an evaluation
If you want to run an evaluation on a dataset, you can use the `EvaluationRunner` class.
```python
from time import perf_counter
from continuous_eval.data_downloader import example_data_downloader
from continuous_eval.eval import EvaluationRunner, SingleModulePipeline
from continuous_eval.eval.tests import GreaterOrEqualThan
from continuous_eval.metrics.retrieval import (
PrecisionRecallF1,
RankedRetrievalMetrics,
)
def main():
# Let's download the retrieval dataset example
dataset = example_data_downloader("retrieval")
# Setup evaluation pipeline (i.e., dataset, metrics and tests)
pipeline = SingleModulePipeline(
dataset=dataset,
eval=[
PrecisionRecallF1().use(
retrieved_context=dataset.retrieved_contexts,
ground_truth_context=dataset.ground_truth_contexts,
),
RankedRetrievalMetrics().use(
retrieved_context=dataset.retrieved_contexts,
ground_truth_context=dataset.ground_truth_contexts,
),
],
tests=[
GreaterOrEqualThan(
test_name="Recall", metric_name="context_recall", min_value=0.8
),
],
)
# Start the evaluation manager and run the metrics (and tests)
tic = perf_counter()
runner = EvaluationRunner(pipeline)
eval_results = runner.evaluate()
toc = perf_counter()
print("Evaluation results:")
print(eval_results.aggregate())
print(f"Elapsed time: {toc - tic:.2f} seconds\n")
print("Running tests...")
test_results = runner.test(eval_results)
pri
```
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/relari-ai-continuous-eval`](/api/graphcanon/tools/relari-ai-continuous-eval)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_
