langwatch

langwatch/langwatch

The platform for LLM evaluations and AI agent testing

GitHub Website

3.3k

Stars

327

Forks

668

Open issues

Watchers

TypeScript Apache-2.0Last pushed Jul 7, 2026

Overview

LangWatch provides tools to test, simulate, evaluate, and monitor language model-powered agents in end-to-end scenarios before release and during production. This includes features such as end-to-end agent simulations, real-time eval + observability, dataset creation, evaluation, and prompt optimization.

Similar tools

ollama

ollama/ollama

Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

176k 17kGo

prompts.chat

f/prompts.chat

The world's largest open-source prompt library for AI

165k 21kHTML

transformers

huggingface/transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models

162k 34kPython

langflow

langflow-ai/langflow

Langflow is a powerful platform for building and deploying AI-powered agents and workflows.

151k 9.5kPython

open-webui

open-webui/open-webui

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

145k 21kPython

awesome-llm-apps

Shubhamsaboo/awesome-llm-apps

100+ AI Agent & RAG apps you can actually run — clone, customize, ship.

117k 17kPython

Install

npm install langwatch

npm Docs / website

README

Website · Docs · Discord · Self-hosting

Why LangWatch?

The platform for LLM evaluations and AI agent testing. We help teams test, simulate, evaluate, and monitor LLM-powered agents end-to-end — before release and in production. Built for teams that need regression testing, simulations, and production observability without building custom tooling.

End-to-end agent simulations Run realistic scenarios against your full stack (tools, state, user simulator, judge) and pinpoint where your agents break, and why? down to each decision.
Eval + observability + prompts in one loop Trace → dataset → evaluate → optimize prompts/models → re-test. No glue code, no tool sprawl.
Open standards, no lock-in OpenTelemetry/OTLP-native. Framework- and LLM-provider agnostic by design.
AI Gateway for governance + cost control OpenAI/Anthropic-compatible proxy with virtual keys, hierarchical budgets, inline guardrails, automatic fallback across providers, and Anthropic cache_control passthrough. ~700 ns hot-path overhead. Ships as a separate Go binary (services/gateway/) + Helm sub-chart (charts/gateway/).
Collaboration that doesn't slow shipping Review runs, annotate failures, and ship fixes faster. Let domain experts label edge cases with annotations & queues, keep prompts in Git with the GitHub integration, and link prompt versions to traces.

LangWatch gives you full visibility into agent behavior and the tools to systematically improve reliability, performance, and cost, while keeping you in control of your AI system