Home/Categories/Evaluation & Observability

Category · 440 tools

Evaluation & Observability

Tracing, evaluation, monitoring, and observability for LLM and agent systems — measuring quality, cost, and latency (Langfuse, Phoenix, OpenLIT).

GraphCanon updated today · 151 views this month

440

Tools

9

Languages

114k

Leader stars

generative-ai-for-beginners

Top tool

Featured comparisons in Evaluation & Observability

MaxKB vs WeKnora activepieces vs coze-loop gateway vs gateway AgentGuide vs JavaGuide ECC vs intellagent agenta vs coze-loop agenta vs paddler agenta vs langfuse

All comparisons →

Stacks using Evaluation & Observability

The RAG stackstack →The AI agent stackstack →

Tools in this category

The highest-adoption tools in Evaluation & Observability, linked as a neighbourhood. Follow any node to keep traversing.

generative-ai-for-beginners114k★netdata80k★headroom66k★TrendRadar61k★CL4R1T4S46k★LibreChat41k★prompt-optimizer33k★langfuse32k★gitleaks29k★

Showing the top 60 of 440.

generative-ai-for-beginners

21 Lessons for Getting Started with Generative AI

Very activeJupyter Notebookaiazurepushed 4w

headroom

Compress tool outputs and data to reduce tokens before reaching the LLM.

Very activePythonagentaipushed 5d

CL4R1T4S

Leaked system prompts for various AI agents include ChatGPT, Claude, Gemini among others emphasizing transparency and access.

Very activeagentsaipushed 3w

LibreChat

Enhanced ChatGPT Clone with extensive features and integrations for self-hosting

Very activeTypeScriptaianthropicpushed 3w

mlflow

AI engineering platform for debugging, evaluating, monitoring, and optimizing AI applications

Very activePythonagentopsagentspushed 1d

langfuse

Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets

Very activeTypeScriptanalyticsautogenpushed 2w

Anthropic-Cybersecurity-Skills

817 structured cybersecurity skills for AI agents

ActivePythonai-agentsclaude-codepushed 1w

pentagi

Fully autonomous AI Agents system for complex penetration testing tasks

ActiveGoai-agentsai-security-toolpushed 2w

promptfoo

Tool for evaluating prompts and AI agents by comparing performance across various models and red teaming.

Very activeTypeScriptcici-cdpushed 2w

WeKnora

Open-source LLM knowledge platform for creating a queryable RAG, autonomous reasoning agent, and self-maintaining Wiki.

Very activeGoagentagenticpushed 5d

opik

Debug, evaluate, and monitor your LLM applications with comprehensive tracing and production-ready dashboards

Very activePythonevaluationhacktoberfestpushed 1w

ai-guide

免费开放的AI知识共享平台

ActiveJavaScriptaiartificial-intelligencepushed 1w

mastra

Modern TypeScript framework for AI-powered applications and agents

Very activeTypeScriptagentsaipushed 4d

prompt-optimizer

An AI prompt optimizer for writing better prompts and getting better AI results.

Very activeTypeScriptai-promptsai-toolspushed 1w

FinGPT

FinGPT: Open-Source Financial Large Language Models

ActiveJupyter Notebookchatgptfinancepushed 2w

ai-berkshire

AI-era Berkshire: a value investing research framework utilizing Claude Code / Codex with methodologies from Warren Buffett, Charlie Munger among others and multi-Agent adversarial analysis.

Very activePythonaiai-agentpushed 3w

bisheng

BISHENG is an open LLM devops platform for next generation Enterprise AI applications

Very activePythonagentaipushed 2d

TrendRadar

AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.

ActivePythonaibarkpushed 1mo

gitleaks

Find secrets with Gitleaks 🔑

ActiveGoai-poweredci-cdpushed 3w

open-swe

An Open-Source Asynchronous Coding Agent

Very activePythonagentagentspushed 1d

phoenix

AI Observability & Evaluation

Very activePythonagentsai-monitoringpushed 2w

casdoor

An open-source Agent-first Identity and Access Management (IAM) / LLM MCP & agent gateway and auth server

Very activeGoagentagentic-aipushed 2d

awesome-LLM-resources

Summary of the world's best LLM resources.

Very activeawesome-listbookpushed 6d

oumi

Easily fine-tune, evaluate and deploy open source LLMs/VLMs

Very activePythondpoevaluationpushed 4w

evolver

The GEP-powered self-evolving engine for AI agents.

Very activeJavaScripta2aagent-evolutionpushed 3w

AgentGuide

AI Agent development guide with career and interview preparation

ActiveHTMLagenticragai-agentpushed 2w

superagent

Superagent SDK

Very activeTypeScriptaianthropicpushed 1w

manifest

Connect Your Agents And Harnesses With Any Provider

Very activeTypeScriptaiai-pushed 1w

plano

An AI-native proxy and data plane for agentic apps

Very activeRustai-gatewayai-gateway-supportpushed 1d

coze-loop

Next-generation AI Agent Optimization Platform

Very activeGoagentagent-evaluationpushed today

helicone

Open source LLM observability platform

Very activeTypeScriptagent-monitoringanalyticspushed 4d

zenml

One AI Platform from Pipelines to Agents

Very activePythonagentopsagentspushed today

heretic

Fully automatic censorship removal for language models

Very activePythonabliterationllmpushed 6d

whichllm

Command-line tool to find and benchmark local LLM performance

ActivePythonaiapple-siliconpushed 2w

deepeval

LLM Evaluation Framework.

Very activePythonevaluation-frameworkevaluation-metricspushed 3w

VLMEvalKit

An open-source evaluation toolkit for large vision-language models

Very activePythonchatgptclaudepushed 3d

mcp-context-forge

AI Gateway and registry for MCP, A2A, REST/gRPC APIs

Very activePythonagentsaipushed 3w

agenta

The open-source LLMOps platform for prompt management, evaluation, and observability.

Very activeTypeScriptagent-builderagent-observabilitypushed 1w

Awesome-Multimodal-Large-Language-Models

Latest Advances on Multimodal Large Language Models

Very activechain-of-thoughtin-context-learningpushed 6d

AutoRAG

Open-source framework for RAG evaluation and optimization via AutoML

Very activeTypeScriptanalysisautomlpushed 2w

ruoyi-ai

一站式AI应用开发框架

Very activeJavaagentaipushed 2w

LLMForEverybody

LLM knowledge sharing for everyone, essential reading before big model interviews

Very activeJupyter Notebookagentinterview-practicepushed 4d

Kiln

Build, Evaluate, and Optimize AI Systems

Very activePythonaichain-of-thoughtpushed 4w

Auto-claude-code-research-in-sleep

Lightweight Markdown-only skills for autonomous ML research

Very activePythonai-researchai-toolspushed 1mo

netdata

The fastest path to AI-powered full stack observability for lean teams

Very activeGoaialertingpushed 3w

gateway

A high-performance AI Gateway connecting to over 1,600 LLMs with guardrails.

SteadyTypeScriptai-gatewaygatewaypushed 2mo

AI-Infra-Guard

A full-stack AI Red Teaming platform securing AI ecosystems

Very activePythonagentagent-securitypushed 3w

doris

Real-time analytics and hybrid search database for AI agents

Very activeC++agentaipushed 2d

lamda

Android full-stack device control platform with remote access and automation capabilities

Very activePythonadbagentspushed 1w

awesome-chatgpt-zh

A comprehensive guide to better utilizing ChatGPT in Chinese.

ActivePythonagiaipushed 1mo

giskard-oss

Open-Source Evaluation & Testing library for LLM Agents

Very activePythonagent-evaluationai-red-teampushed 2w

aim

An easy-to-use & supercharged open-source experiment tracker

Very activePythonaidata-sciencepushed 3w

langwatch

The platform for LLM evaluations and AI agent testing

Very activeTypeScriptaianalyticspushed 1w

Guardrails

Open-source toolkit for adding programmable guardrails to LLM-based conversational systems

Very activePythonagentsgenerative-aipushed 1w

ChatGPT-Shortcut

Maximize your efficiency and productivity through prompt management, customization, and sharing.

Very activeTypeScriptaiai-toolspushed 1mo

owl

Optimized Workforce Learning for General Multi-Agent Assistance

Very activePythonagentartificial-intelligencepushed 6d

logfire

AI observability platform for production LLM and agent systems

Very activePythonagent-observabilityaipushed 1w

tracecat

Open-source security automation platform for teams and AI agents

Very activePythonagentsautomationpushed 1w

dagster

An orchestration platform for data assets

Very activePythonanalyticsdagsterpushed 1w

trulens

Evaluation and Tracking for LLM Experiments and AI Agents

Very activePythonagent-evaluationagentopspushed today

Common questions

What are the best evaluation & observability tools?: GraphCanon ranks Evaluation & Observability tools by GitHub adoption and freshness. generative-ai-for-beginners is the current leader (113,577 stars). See the full list on this page - sorted by stars, with maintenance labels and graph relationships.
How does GraphCanon rank Evaluation & Observability tools?: We sort by GitHub stars and push recency on category pages, not paid placement. Alternatives and compare pages use typed graph edges (alternative, successor, integrates_with) plus shared categories - constraint-first, not marketing votes.
How many tools are in Evaluation & Observability?: 440 published tools are tagged with Evaluation & Observability in the GraphCanon knowledge graph.
What are popular Evaluation & Observability comparisons?: Head-to-head compare pages in this category include MaxKB vs WeKnora, activepieces vs coze-loop, gateway vs gateway. Each comparison uses live GitHub stats and optional trust signals - see the comparisons block on this page.
Which stacks use Evaluation & Observability?: Curated workflow pages that include Evaluation & Observability: The RAG stack; The AI agent stack. Each stack step includes when-not-to-use guidance.
Is there a machine-readable Evaluation & Observability list?: Yes. Append .md to this URL or fetch `/md/categories/evaluation-observability` for a markdown twin. The JSON API exposes the same corpus at `/api/graphcanon/categories/evaluation-observability`.

Was this helpful?

Anonymous feedback helps us improve pages and translations.