{"data":{"slug":"tolitius-cupel","name":"cupel","tagline":"discovery tool for evaluating LLM performance","github_url":"https://github.com/tolitius/cupel","owner":"tolitius","repo":"cupel","owner_avatar_url":"https://avatars.githubusercontent.com/u/136575?v=4","primary_language":"Python","stars":64,"forks":0,"topics":["llm","llm-evaluation","local-llm"],"archived":false,"github_pushed_at":"2026-08-31T04:05:52+00:00","maintenance_label":"Steady","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/tolitius-cupel","markdown_url":"https://www.graphcanon.com/tools/tolitius-cupel.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/tolitius-cupel","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=tolitius-cupel","description":"discover LLMs punching above their weight","homepage_url":"https://cupel.run","license":"Apache-2.0","open_issues":2,"watchers":0,"ai_summary":"Cupel is a JavaScript-based toolkit for discovering and evaluating the capabilities of various LLMs through configurable prompt generation, scoring mechanisms, multi-turn dialogues, and local inference server discovery.","readme_excerpt":"## install\n\n```bash\ncurl -fsSL https://cupel.run/install | bash\n```\nor\n```\npip install cupel\n```\nthe UI is bundled in the package\n\n---\n\n## quick start\n\n```bash\ncupel\n```\n\nopens a browser at `localhost:8042`\n\n<img src=\"doc/first-time.png\" alt=\"first time cupel is started\" width=\"76%\">\n\nships with example data (8 models scored by Claude Opus 4.6 on 8 prompts) — the dashboard is populated on first launch\n\n- **LLM-assisted authoring** — describe what you want to test, an LLM drafts the prompt and 0–3 rubric\n- **local + cloud** — oMLX, Ollama, LM Studio, SGLang, OpenRouter, Anthropic, OpenAI\n- **configurable judge** — any model can score responses on a 0–3 rubric with reasoning\n- **thinking model support** — separates `<think>` blocks from answers, only judges the response\n- **multi-turn + tool calling** — multi-step conversations with injected tool results\n- **speed tracking** — tok/s and response times per model\n- **auto-discovery** — probes known ports for local inference servers\n\n---\n\n## license\n\nCopyright © 2026 tolitius\n\nDistributed under the [Apache 2.0](LICENSE) License.","github_created_at":"2026-04-08T07:57:39+00:00","created_at":"2026-07-15T10:40:43.215679+00:00","updated_at":"2026-10-19T06:00:23.200622+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"inference-servers-discovery","name":"inference-servers-discovery"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"local-llm","name":"local-llm"},{"slug":"multi-turn-dialogue","name":"multi-turn-dialogue"},{"slug":"prompt-generation","name":"prompt-generation"},{"slug":"scoring-mechanisms","name":"scoring-mechanisms"}],"trust":{"provenance":{"is_fork":false,"github_id":1204659735,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-19T06:00:21.837Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":0,"days_since_push":49,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"last_scan_at":"2026-07-15T10:40:45.718071+00:00","scan_profile":"deps","status":"no_lockfile","critical_count":0,"high_count":0,"medium_count":0,"low_count":0,"scanner":"osv@v1","scan_id":"bc5c4e55-366c-4e8c-83c6-33d2c1405c9a","pending_review_count":0}},"capability_facts":{"mcp":{"method":"github-tree-and-contents@2","reason":"No explicit MCP server declaration was found in the inspected manifests. Other layouts and implementations were not tested.","source":"repo_scan","status":"unknown","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.243Z","server_manifest":null},"scan":{"method":"github-tree-and-contents@2","source":"repo_scan","status":"complete","version":2,"coverage":{"scope":"bounded-repository-manifests","tree_complete":true,"selected_files":1,"candidate_files":1},"revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.243Z","observations":[{"path":"@revision","bytes":4633,"status":"present","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/commit/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","http_status":200,"observed_at":"2026-10-19T06:00:22.436Z"},{"path":"@tree","bytes":16456,"status":"present","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","http_status":200,"observed_at":"2026-10-19T06:00:22.652Z"},{"path":"docker-compose.yml","reason":"not-listed-in-complete-tree","status":"not-found","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.652Z"},{"path":"Dockerfile","reason":"not-listed-in-complete-tree","status":"not-found","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.652Z"},{"path":"mcp.json","reason":"not-listed-in-complete-tree","status":"not-found","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.652Z"},{"path":"package.json","reason":"not-listed-in-complete-tree","status":"not-found","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.652Z"},{"path":"pyproject.toml","bytes":1128,"status":"present","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/blob/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a/pyproject.toml","http_status":200,"observed_at":"2026-10-19T06:00:22.817Z"},{"path":"server.json","reason":"not-listed-in-complete-tree","status":"not-found","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.652Z"}]},"has_cli":{"value":true,"method":"repository-declaration@2","source":"pyproject.toml:[project.scripts]","status":"declared","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/blob/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a/pyproject.toml","observed_at":"2026-10-19T06:00:22.243Z"},"languages":{"value":["python"],"method":"github-tree-and-contents@2","source":"github.language+pyproject.toml","revision":"1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","source_url":"https://github.com/tolitius/cupel/tree/1d632392ff5314d573ba2b3d0f589fa9e3d2e35a","observed_at":"2026-10-19T06:00:22.243Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-10-19T06:00:22.243Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When aiming to evaluate LLMs on local servers due to its auto-discovery feature for known ports of inference servers","For generating initial test scenarios where an LLM can draft prompts and rubrics based on input descriptions"],"when_not_to_use":["If you require a solution that supports a non-JavaScript runtime environment, as Cupel is JavaScript-exclusive","When you need a tool without UI capabilities since Cupel's UI is bundled in the package and may not suit headless operations"],"source":"enrich:decision_facts","observed_at":"2026-07-17T05:24:03.412Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Unverified catalog guidance: Cupel is a JavaScript-based toolkit for discovering and evaluating the performance of large language models using configurable prompts, scoring mechanisms, multi-turn dialogues, and local inference server discovery."}]}}