litmus
Enrichment pendingLitmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and
GraphCanon updated today · GitHub synced today
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (107d since push)
- As of today
- Provenance
- Not a fork · Organization account
- As of today
- Security (OSV)
- No lockfile
- As of today
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for Google. Display-only - separate from trust and ranking.
- Company
- Google·GitHub org profile·3d
- Employees
- 47,756·Wikidata (P1128 employees)·3d
- Commercial model
- Pure OSS·GitHub org profile (public repos)·3d
Install
git clone https://github.com/google/litmusSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and assessing the performance of your LLM-powered applications.
Capability facts
- Languages
- vue
Source: github.language · Jul 15, 2026
Categories
Graph entities
Tags
README
Getting Started
1. Quick Deployment with the Litmus CLI:
-
This is the easiest way to set up Litmus.
-
Make sure you have the Google Cloud SDK installed and configured with the correct project.
-
Install the Litmus CLI:
- Linux:
- install:
curl https://storage.googleapis.com/litmus-cloud/install/linux.sh | sudo sh - binary: https://storage.googleapis.com/litmus-cloud/prod/linux/litmus
- sha256: https://storage.googleapis.com/litmus-cloud/prod/linux/litmus.sha256
- install:
- OSX:
- install:
curl https://storage.googleapis.com/litmus-cloud/install/osx.sh | sudo sh - binary: https://storage.googleapis.com/litmus-cloud/prod/osx/litmus
- sha256: https://storage.googleapis.com/litmus-cloud/prod/osx/litmus.sha256
- install:
- Linux:
-
Deploy Litmus:
litmus deploy -
Deploy the proxy service (optional):
litmus proxy deploy -
The deployment script will:
- Create service accounts and grant necessary permissions.
- Deploy the worker service and the API service to Cloud Run.
- Deploy the proxy service to Cloud Run if you choose to.
2. Manual Setup:
- If you prefer manual deployment:
- Set up your Google Cloud project: Enable the required APIs (Firestore, Cloud Run, BigQuery).
- Deploy the worker service: Build a Docker image for the worker service in the
workerdirectory and deploy it to Cloud Run. - Deploy the API service: Build a Docker image for the API service in the
apidirectory and deploy it to Cloud Run. - Deploy the proxy service: Build a Docker image for the proxy service in the
proxydirectory and deploy it to Cloud Run. - Configure API settings: Create a
api/util/settings.pyfile with your Google Cloud project ID, region, and other settings. - Deploy the UI: Deploy the user interface code in the
api/uidirectory to a web server (e.g., Nginx, Apache). - Connect the UI: Configure the UI to connect to the deployed API service.
3. Using Litmus:
- Access the web interface.
- Create and manage test templates, defining test requests, expected responses, and LLM evaluation prompts.
- Select your desired evaluation methods in your templates (Custom LLM Evaluation, Ragas, DeepEval).
- Optionally configure your LLM client to use the proxy service.
- Submit test runs, monitor progress, and analyze the detailed results, including LLM-based assessments.
- Explore proxy data and understand your LLM usage patterns.
License
Apache 2.0; see LICENSE for details.
For agents
This page has a .md twin and JSON over the API.