Supercharge Your LLM Application Evaluations 🚀
Documentation | Quick start | Join Discord | Blog | NewsLetter | Careers
Objective metrics, intelligent test generation, and data-driven insights for LLM apps Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready? We also do production-aligned test set generation. ## Key Features - 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics. - 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios. - 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools. - 📊 Build feedback loops: Leverage production data to continually improve your LLM applications. ## :shield: Installation Pypi: ```bash pip install ragas ``` Alternatively, from source: ```bash pip install git+https://github.com/vibrantlabsai/ragas ``` ## :fire: Quickstart ### Clone a Complete Example Project The fastest way to get started is to use the `ragas quickstart` command: ```bash # List available templates ragas quickstart # Create a RAG evaluation project ragas quickstart rag_eval # Specify where you want to create it. ragas quickstart rag_eval -o ./my-project ``` Available templates: - `rag_eval` - Evaluate RAG systems Coming Soon: - `agent_evals` - Evaluate AI agents - `benchmark_llm` - Benchmark and compare LLMs - `prompt_evals` - Evaluate prompt variations - `workflow_eval` - Evaluate complex workflows ### Evaluate your LLM App `ragas` comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using `DiscreteMetric`: ```python import asyncio from openai import AsyncOpenAI from ragas.metrics import DiscreteMetric from ragas.llms import llm_factory # Setup your LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o", client=client) # Create a custom aspect evaluator metric = DiscreteMetric( name="summary_accuracy", allowed_values=["accurate", "inaccurate"], prompt="""Evaluate if the summary is accurate and captures key information. Response: {response} Answer with only 'accura ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/vibrantlabsai-ragas`](/api/graphcanon/tools/vibrantlabsai-ragas) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_