{"data":{"slug":"brontoguana-krasis","name":"krasis","tagline":"Hybrid LLM Runtime for Efficient Large Model Inference on Consumer Hardware","github_url":"https://github.com/brontoguana/krasis","owner":"brontoguana","repo":"krasis","owner_avatar_url":"https://avatars.githubusercontent.com/u/7049825?v=4","primary_language":"C++","stars":521,"forks":33,"topics":["cpu-inference","gguf-model-support","gpu-inference","high-performance-inference","hybrid-inference","inference-engine","inference-optimization","large-language-models","llama-cpp-alternative","llm-inference","mixture-of-experts","transformer"],"archived":false,"github_pushed_at":"2026-09-15T07:46:41+00:00","maintenance_label":"Very active","stars_delta_30d":37,"url":"https://www.graphcanon.com/tools/brontoguana-krasis","markdown_url":"https://www.graphcanon.com/tools/brontoguana-krasis.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/brontoguana-krasis","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=brontoguana-krasis","description":"Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware","homepage_url":null,"license":"Other","open_issues":19,"watchers":10,"ai_summary":"Krasis is designed to enable efficient running of larger language models with constraints on consumer-grade VRAM. It supports high-performance inference, hybrid CPU-GPU execution, and is an alternative to existing solutions such as llama-cpp.","readme_excerpt":"## Install\n\nThe current release is\n[`v1.0.16`](https://github.com/brontoguana/krasis/releases/tag/v1.0.16).\n\n**Native Windows:** [Download the Krasis Windows installer](https://github.com/brontoguana/krasis/releases/download/v1.0.16/KrasisSetup-1.0.16-win64.exe).\nIt installs Krasis for the current user, includes its own Python runtime, and\nadds `Krasis` to the Start Menu.\n\n**Linux or WSL2:**\n\n```bash\ncurl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- prerelease\n```\n\nSee [all releases](https://github.com/brontoguana/krasis/releases) for older\nversions and individual wheel/source assets.\n\nYou can [contact me here](https://forms.gle/ue4nvyvNNHtUZ7MQ7), but for bugs,\nsetup problems, model requests, or feature requests please open a GitHub issue.\n\nIf you want to monitor Krasis during runs, [check out ktop](https://github.com/brontoguana/ktop).\n\n---\n\n## Tradeoffs And Requirements\n\n- Krasis currently targets NVIDIA GPUs with CUDA, including Ampere and newer\n  architectures. The production HQQ attention and compact KV cache modes do not\n  require FP8 support.\n- Input models should be BF16 safetensors from Hugging Face or another local\n  safetensors source.\n- First run is slower because Krasis builds optimized local caches. Later runs\n  reuse those caches.\n- Disk usage must cover the source model plus Krasis cache artifacts under\n  `~/.krasis`.\n- System RAM should be sized for the selected quantized cache and HCS backing\n  store. Larger models need substantial RAM even when GPU VRAM is limited.\n- Production runs should use quantized INT4/INT8 expert caches and HQQ\n  attention. BF16-heavy modes are validation/debug modes, not normal deployment\n  targets.\n\n---\n\n### Requirements\n\n- Native x86-64 Windows, Linux (including Ubuntu 24.04+), or WSL2\n- Python 3.10+ on Linux/WSL; native Windows uses the release-pinned private\n  Python included by the installer\n- NVIDIA GPU with CUDA drivers installed\n- Rust is only needed for source builds, not normal wheel installs\n- Enough disk/RAM for the source model and generated Krasis caches\n\n---\n\n### 1. Install Krasis\n\nLinux/WSL:\n\n```bash\ncurl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- prerelease\n```\n\nThis creates a managed environment at `~/.krasis/venv`, installs Krasis,\nsymlinks commands into `~/.local/bin`, and updates PATH for the current shell.\nNo sudo is required for the Krasis install itself. Omit `prerelease` when\ninstalling the latest stable release.\n\nNative Windows:\n\n[Download `KrasisSetup-1.0.16-win64.exe`](https://github.com/brontoguana/krasis/releases/download/v1.0.16/KrasisSetup-1.0.16-win64.exe).\nThe installer creates a per-user install under\n`%LOCALAPPDATA%\\Programs\\Krasis`, installs and validates a release-pinned\nprivate Python/Krasis/PyTorch runtime, and adds `Krasis` and `Krasis Manager`\nshortcuts to the Start Menu folder. It never uses or modifies a system Python.\n`Krasis` opens the native interactive launcher in a maximized, resizable\nconsole; `Krasis Manager` starts the localhost management dashboard. Models\nand caches still live under\n`%USERPROFILE%\\.krasis`. The first install downloads the pinned CUDA PyTorch\nwheel and can take several minutes.\n\nNative Windows packages Marlin, FlashAttention, and FLA sidecars for supported\nAmpere and newer NVIDIA architectures.\n\n---\n\n### 2. Install CUDA Dependencies\n\n```bash\nkrasis-setup\n```\n\nThis installs runtime CUDA/PyTorch dependencies when needed. It is usually only\nrequired once per machine.\n\n---\n\n## License\n\nSSPL-1.0\n\nKrasis is free to use, modify, and distribute.\n\nIf you want to support the project or offer Krasis as part of a commercial\nproduct or a hosted/managed service, please [get in touch](https://forms.gle/ue4nvyvNNHtUZ7MQ7).","github_created_at":"2026-02-09T17:54:07+00:00","created_at":"2026-07-11T11:45:50.492118+00:00","updated_at":"2026-09-20T06:35:54.145718+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cpu-inference","name":"cpu-inference"},{"slug":"gguf-model-support","name":"gguf-model-support"},{"slug":"gpu-inference","name":"gpu-inference"},{"slug":"high-performance-inference","name":"high-performance-inference"},{"slug":"hybrid-inference","name":"hybrid-inference"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"mixture-of-experts","name":"mixture-of-experts"},{"slug":"transformer","name":"transformer"}],"trust":{"provenance":{"is_fork":false,"github_id":1153822578,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T03:45:16.648Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":27,"days_since_push":4,"last_release_at":"2026-09-15T07:46:42Z","stars_delta_30d":37,"open_issues_delta_30d":11},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:45:51.791Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T03:45:17.905Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-20T03:45:17.905Z"},"languages":{"value":["c++","python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T03:45:17.905Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-09-20T03:45:17.905Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When aiming for efficient operation of larger language models with limited VRAM, as Krasis optimizes memory utilization specifically to support this scenario.","- If you seek an alternative runtime that can handle the latest GGUF-supported models without significant compromises in performance compared to other toolkits like llama-cpp."],"when_not_to_use":["- Avoid using Krasis if your hardware setup does not include both CPU and GPU capabilities, as its hybrid execution relies on utilizing both components for optimal performance.","- If you prioritize running lightweight models with minimal memory footprint on low-end devices, Krasis might not be the ideal choice given it is optimized for larger-scale model inference."],"source":"enrich:decision_facts","observed_at":"2026-07-15T08:50:25.085Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Krasis is designed to offer efficient large model inference on consumer-grade hardware through hybrid CPU-GPU execution and high-performance optimization."}]}}