{"data":{"slug":"google-litmus","name":"litmus","tagline":"A comprehensive LLM testing and evaluation tool for GenAI application development with user-friendly UI","github_url":"https://github.com/google/litmus","owner":"google","repo":"litmus","owner_avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=4","primary_language":"Vue","stars":52,"forks":9,"topics":["api","apitesting","cicd","devops","llm","llm-evaluation","llm-security","llm-training","llmops","testing","testing-tools"],"archived":false,"github_pushed_at":"2026-03-29T16:13:21+00:00","maintenance_label":"Slowing","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/google-litmus","markdown_url":"https://www.graphcanon.com/tools/google-litmus.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/google-litmus","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=google-litmus","description":"Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and assessing the performance of your LLM-powered applications. ","homepage_url":"https://google.github.io/litmus/","license":"Apache-2.0","open_issues":5,"watchers":3,"ai_summary":"Litmus is an all-inclusive platform aimed at developing and evaluating the performance of AI applications powered by large language models.","readme_excerpt":"## Getting Started\n\n\n\n**1. Quick Deployment with the Litmus CLI:**\n\n- This is the easiest way to set up Litmus.\n- Make sure you have the Google Cloud SDK installed and configured with the correct project.\n\n- Install the Litmus CLI:\n\n  - **Linux**:\n    - install:`curl https://storage.googleapis.com/litmus-cloud/install/linux.sh | sudo sh`\n    - binary: [https://storage.googleapis.com/litmus-cloud/prod/linux/litmus](https://storage.googleapis.com/litmus-cloud/prod/linux/litmus)\n    - sha256: [https://storage.googleapis.com/litmus-cloud/prod/linux/litmus.sha256](https://storage.googleapis.com/litmus-cloud/prod/linux/litmus.sha256)\n  - **OSX**:\n    - install:`curl https://storage.googleapis.com/litmus-cloud/install/osx.sh | sudo sh`\n    - binary: [https://storage.googleapis.com/litmus-cloud/prod/osx/litmus](https://storage.googleapis.com/litmus-cloud/prod/osx/litmus)\n    - sha256: [https://storage.googleapis.com/litmus-cloud/prod/osx/litmus.sha256](https://storage.googleapis.com/litmus-cloud/prod/osx/litmus.sha256)\n\n- Deploy Litmus:\n  `litmus deploy`\n\n- Deploy the proxy service (optional):\n  `litmus proxy deploy`\n\n- **The deployment script will:**\n  - Create service accounts and grant necessary permissions.\n  - Deploy the worker service and the API service to Cloud Run.\n  - Deploy the proxy service to Cloud Run if you choose to.\n\n**2. Manual Setup:**\n\n- **If you prefer manual deployment:**\n  - **Set up your Google Cloud project:** Enable the required APIs (Firestore, Cloud Run, BigQuery).\n  - **Deploy the worker service:** Build a Docker image for the worker service in the `worker` directory and deploy it to Cloud Run.\n  - **Deploy the API service:** Build a Docker image for the API service in the `api` directory and deploy it to Cloud Run.\n  - **Deploy the proxy service:** Build a Docker image for the proxy service in the `proxy` directory and deploy it to Cloud Run.\n  - **Configure API settings:** Create a `api/util/settings.py` file with your Google Cloud project ID, region, and other settings.\n  - **Deploy the UI:** Deploy the user interface code in the `api/ui` directory to a web server (e.g., Nginx, Apache).\n  - **Connect the UI:** Configure the UI to connect to the deployed API service.\n\n**3. Using Litmus:**\n\n- Access the web interface.\n- Create and manage test templates, defining test requests, expected responses, and LLM evaluation prompts.\n- Select your desired evaluation methods in your templates (Custom LLM Evaluation, Ragas, DeepEval).\n- Optionally configure your LLM client to use the proxy service.\n- Submit test runs, monitor progress, and analyze the detailed results, including LLM-based assessments.\n- Explore proxy data and understand your LLM usage patterns.\n\n---\n\n## License\n\nApache 2.0; see [`LICENSE`](LICENSE) for details.","github_created_at":"2024-06-13T21:33:17+00:00","created_at":"2026-07-15T10:40:50.645156+00:00","updated_at":"2026-10-19T06:00:30.065276+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"api","name":"api"},{"slug":"apitesting","name":"apitesting"},{"slug":"cicd","name":"cicd"},{"slug":"devops","name":"devops"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"testing-tools","name":"testing-tools"}],"trust":{"provenance":{"is_fork":false,"github_id":814860684,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-19T06:00:28.210Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":203,"last_release_at":"2024-09-18T07:33:45Z","stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"last_scan_at":"2026-07-15T10:40:53.03509+00:00","scan_profile":"deps","status":"no_lockfile","critical_count":0,"high_count":0,"medium_count":0,"low_count":0,"scanner":"osv@v1","scan_id":"84597cca-28eb-43be-a3db-658e3c279722","pending_review_count":0}},"capability_facts":{"mcp":{"method":"github-tree-and-contents@2","reason":"No explicit MCP server declaration was found in the inspected manifests. Other layouts and implementations were not tested.","source":"repo_scan","status":"unknown","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.601Z","server_manifest":null},"scan":{"method":"github-tree-and-contents@2","source":"repo_scan","status":"complete","version":2,"coverage":{"scope":"bounded-repository-manifests","tree_complete":true,"selected_files":5,"candidate_files":5},"revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.601Z","observations":[{"path":"@revision","bytes":4822,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/commit/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","http_status":200,"observed_at":"2026-10-19T06:00:28.792Z"},{"path":"@tree","bytes":54955,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","http_status":200,"observed_at":"2026-10-19T06:00:28.944Z"},{"path":"api/Dockerfile","bytes":1400,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/blob/0d1f0aeb20361e778695bffea10a4ed5f73f63ff/api/Dockerfile","http_status":200,"observed_at":"2026-10-19T06:00:29.131Z"},{"path":"api/ui/package.json","bytes":1745,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/blob/0d1f0aeb20361e778695bffea10a4ed5f73f63ff/api/ui/package.json","http_status":200,"observed_at":"2026-10-19T06:00:29.125Z"},{"path":"docker-compose.yml","reason":"not-listed-in-complete-tree","status":"not-found","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.944Z"},{"path":"Dockerfile","reason":"not-listed-in-complete-tree","status":"not-found","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.944Z"},{"path":"docs/package.json","bytes":206,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/blob/0d1f0aeb20361e778695bffea10a4ed5f73f63ff/docs/package.json","http_status":200,"observed_at":"2026-10-19T06:00:29.101Z"},{"path":"mcp.json","reason":"not-listed-in-complete-tree","status":"not-found","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.944Z"},{"path":"package.json","reason":"not-listed-in-complete-tree","status":"not-found","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.944Z"},{"path":"proxy/Dockerfile","bytes":681,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/blob/0d1f0aeb20361e778695bffea10a4ed5f73f63ff/proxy/Dockerfile","http_status":200,"observed_at":"2026-10-19T06:00:29.105Z"},{"path":"pyproject.toml","reason":"not-listed-in-complete-tree","status":"not-found","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.944Z"},{"path":"server.json","reason":"not-listed-in-complete-tree","status":"not-found","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.944Z"},{"path":"worker/Dockerfile","bytes":1074,"status":"present","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/blob/0d1f0aeb20361e778695bffea10a4ed5f73f63ff/worker/Dockerfile","http_status":200,"observed_at":"2026-10-19T06:00:29.305Z"}]},"languages":{"value":["vue","javascript","typescript"],"method":"github-tree-and-contents@2","source":"github.language+docs/package.json+api/ui/package.json","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/tree/0d1f0aeb20361e778695bffea10a4ed5f73f63ff","observed_at":"2026-10-19T06:00:28.601Z"},"has_docker":{"value":true,"method":"repository-declaration@2","source":"dockerfile:proxy/Dockerfile","revision":"0d1f0aeb20361e778695bffea10a4ed5f73f63ff","source_url":"https://github.com/google/litmus/blob/0d1f0aeb20361e778695bffea10a4ed5f73f63ff/proxy/Dockerfile","observed_at":"2026-10-19T06:00:28.601Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-10-19T06:00:28.601Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When deploying on Google Cloud with the need for CLI ease of use","For comprehensive evaluation methods like Custom LLM Evaluation, Ragas, DeepEval","If API testing within CI/CD pipelines is a priority"],"when_not_to_use":["In environments not using Google Cloud services and APIs","When manual setup for each component outweighs potential benefits of automation provided by Litmus CLI"],"source":"enrich:decision_facts","observed_at":"2026-07-16T19:11:25.158Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Unverified catalog guidance: Litmus is an extensive GenAI application development platform for testing and evaluating LLM performance, featuring a user-friendly UI and deployment flexibility via CLI or manual setup."}]}}