An open-source framework to evaluate, test and monitor ML and LLM-powered systems.
Documentation | API Reference | Discord Community | Blog | Twitter | Evidently Cloud
# :bar_chart: What is Evidently? Evidently is an open-source Python library to evaluate, test, and monitor ML and LLM systems—from experiments to production. * 🔡 Works with tabular and text data. * ✨ Supports evals for predictive and generative tasks, from classification to RAG. * 📚 100+ built-in metrics from data drift detection to LLM judges. * 🛠️ Python interface for custom metrics. * 🚦 Both offline evals and live monitoring. * 💻 Open architecture: easily export data and integrate with existing tools. Evidently is very modular. You can start with one-off evaluations or host a full monitoring service. ## 1. Reports and Test Suites **Reports** compute and summarize various data, ML and LLM quality evals. * Start with Presets and built-in metrics or customize. * Best for experiments, exploratory analysis and debugging. * View interactive Reports in Python or export as JSON, Python dictionary, HTML, or view in monitoring UI. Turn any Report into a **Test Suite** by adding pass/fail conditions. * Best for regression testing, CI/CD checks, or data validation. * Zero setup option: auto-generate test conditions from the reference dataset. * Simple syntax to set test conditions as `gt` (greater than), `lt` (less than), etc. | Reports | |--| || ## 2. Monitoring Dashboard **Monitoring UI** service helps visualize metrics and test results over time. You can choose: * Self-host the open-source version. [Live demo](https://demo.evidentlyai.com). * Sign up for [Evidently Cloud](https://www.evidentlyai.com/register) (Recommended). Evidently Cloud offers a generous free tier and extra features like dataset and user management, alerting, and no-code evals. [Compare OSS vs Cloud](https://docs.evidentlyai.com/faq/oss_vs_cloud). | Dashboard | |--| || # :woman_technologist: Install Evidently To install from PyPI: ```sh pip install evidently ``` To install Evidently using the Conda installer, run: ```sh conda install -c conda-forge evidently ``` # :arrow_forward: Getting started ## Reports ### LLM evals > This is a simple Hello World. Check the Tutorials for more: [LLM evaluation](https://docs.evidentlyai.com/quickstart_llm). Import the necessary components: ```python import pandas as pd from evidently import Report from evidently import Dataset, DataDefinition from evidently.descriptors import Sentiment, TextLength, Contains from evidently.presets import TextEvals ``` Create a toy dataset with questions and answers. ```python eval_df = pd.DataFrame([ ["What is the capital of Japan?", "The capital of Japan is Tokyo."], ["Who painted the Mona Lisa?", "Leonardo da Vinci."], ["Can you write an essay?", "I'm sorry, but I can't assist with homework."]], columns=["question", "answer"]) ``` Create an Evidently Dataset object and add `descriptors`: row-level evaluators. We'll check for sentiment of each response, its length and whether it contains words indicative of deni ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/evidentlyai-evidently`](/api/graphcanon/tools/evidentlyai-evidently) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_