{"data":{"slug":"google-litmus","name":"litmus","tagline":"Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and","github_url":"https://github.com/google/litmus","owner":"google","repo":"litmus","owner_avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=4","primary_language":"Vue","stars":50,"forks":9,"topics":["api","apitesting","cicd","devops","llm","llm-evaluation","llm-security","llm-training","llmops","testing","testing-tools"],"archived":false,"github_pushed_at":"2026-03-29T16:13:21+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/google-litmus","markdown_url":"https://www.graphcanon.com/tools/google-litmus.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/google-litmus","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=google-litmus","description":"Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and assessing the performance of your LLM-powered applications. ","homepage_url":"https://google.github.io/litmus/","license":"Apache-2.0","open_issues":5,"watchers":3,"ai_summary":null,"readme_excerpt":"## Getting Started\n\n\n\n**1. Quick Deployment with the Litmus CLI:**\n\n- This is the easiest way to set up Litmus.\n- Make sure you have the Google Cloud SDK installed and configured with the correct project.\n\n- Install the Litmus CLI:\n\n  - **Linux**:\n    - install:`curl https://storage.googleapis.com/litmus-cloud/install/linux.sh | sudo sh`\n    - binary: [https://storage.googleapis.com/litmus-cloud/prod/linux/litmus](https://storage.googleapis.com/litmus-cloud/prod/linux/litmus)\n    - sha256: [https://storage.googleapis.com/litmus-cloud/prod/linux/litmus.sha256](https://storage.googleapis.com/litmus-cloud/prod/linux/litmus.sha256)\n  - **OSX**:\n    - install:`curl https://storage.googleapis.com/litmus-cloud/install/osx.sh | sudo sh`\n    - binary: [https://storage.googleapis.com/litmus-cloud/prod/osx/litmus](https://storage.googleapis.com/litmus-cloud/prod/osx/litmus)\n    - sha256: [https://storage.googleapis.com/litmus-cloud/prod/osx/litmus.sha256](https://storage.googleapis.com/litmus-cloud/prod/osx/litmus.sha256)\n\n- Deploy Litmus:\n  `litmus deploy`\n\n- Deploy the proxy service (optional):\n  `litmus proxy deploy`\n\n- **The deployment script will:**\n  - Create service accounts and grant necessary permissions.\n  - Deploy the worker service and the API service to Cloud Run.\n  - Deploy the proxy service to Cloud Run if you choose to.\n\n**2. Manual Setup:**\n\n- **If you prefer manual deployment:**\n  - **Set up your Google Cloud project:** Enable the required APIs (Firestore, Cloud Run, BigQuery).\n  - **Deploy the worker service:** Build a Docker image for the worker service in the `worker` directory and deploy it to Cloud Run.\n  - **Deploy the API service:** Build a Docker image for the API service in the `api` directory and deploy it to Cloud Run.\n  - **Deploy the proxy service:** Build a Docker image for the proxy service in the `proxy` directory and deploy it to Cloud Run.\n  - **Configure API settings:** Create a `api/util/settings.py` file with your Google Cloud project ID, region, and other settings.\n  - **Deploy the UI:** Deploy the user interface code in the `api/ui` directory to a web server (e.g., Nginx, Apache).\n  - **Connect the UI:** Configure the UI to connect to the deployed API service.\n\n**3. Using Litmus:**\n\n- Access the web interface.\n- Create and manage test templates, defining test requests, expected responses, and LLM evaluation prompts.\n- Select your desired evaluation methods in your templates (Custom LLM Evaluation, Ragas, DeepEval).\n- Optionally configure your LLM client to use the proxy service.\n- Submit test runs, monitor progress, and analyze the detailed results, including LLM-based assessments.\n- Explore proxy data and understand your LLM usage patterns.\n\n---\n\n## License\n\nApache 2.0; see [`LICENSE`](LICENSE) for details.","github_created_at":"2024-06-13T21:33:17+00:00","created_at":"2026-07-15T10:40:50.645156+00:00","updated_at":"2026-07-15T10:40:53.390467+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"api","name":"api"},{"slug":"apitesting","name":"apitesting"},{"slug":"cicd","name":"cicd"},{"slug":"devops","name":"devops"},{"slug":"llm","name":"llm"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"llm-security","name":"llm-security"},{"slug":"llm-training","name":"llm-training"}],"trust":{"provenance":{"is_fork":false,"github_id":814860684,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-15T10:40:51.612Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":107,"last_release_at":"2024-09-18T07:33:45Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T10:40:52.015Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-15T10:40:51.379Z"},"languages":{"value":["vue"],"source":"github.language","observed_at":"2026-07-15T10:40:51.379Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-15T10:40:51.379Z"}}}}