{"data":{"slug":"flashinfer-ai-flashinfer","name":"flashinfer","tagline":"FlashInfer is a kernel library for serving large language models","github_url":"https://github.com/flashinfer-ai/flashinfer","owner":"flashinfer-ai","repo":"flashinfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/145061914?v=4","primary_language":"Cuda","stars":6452,"forks":1472,"topics":["attention","cuda","distributed-inference","gpu","jit","large-large-models","llm-inference","moe","nvidia","pytorch"],"archived":false,"github_pushed_at":"2026-09-19T01:31:07+00:00","maintenance_label":"Very active","stars_delta_30d":428,"url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer","markdown_url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/flashinfer-ai-flashinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=flashinfer-ai-flashinfer","description":"FlashInfer: Kernel Library for LLM Serving","homepage_url":"https://flashinfer.ai","license":"Apache-2.0","open_issues":953,"watchers":56,"ai_summary":"A Python-based framework focused on optimizing inference for large-scale language models using CUDA and GPU support.","readme_excerpt":"### Installation\n\n**Quickstart:**\n\n```bash\npip install flashinfer-python\n```\n\n**Package Options:**\n\n- **flashinfer-python**: Core package that compiles/downloads kernels on first use\n- **flashinfer-cubin**: Pre-compiled kernel binaries for all supported GPU architectures\n- **flashinfer-jit-cache**: CUDA-specific shim that installs architecture-specific pre-built kernel providers\n\n**For faster initialization and offline usage**, install the optional packages to have most kernels pre-compiled:\n\n```bash\npip install flashinfer-python\nflashinfer install-cubin-wheel\nflashinfer install-jit-cache-wheel\n```\n\n**For Blackwell (SM100+) CuTe DSL kernels**, install with the CUDA 13 extra to enable Blackwell-optimized kernels:\n\n```bash\npip install flashinfer-python[cu13]\n```\n\n---\n\n### Verify Installation\n\n```bash\nflashinfer show-config\n```\n\n---\n\n### Install from Source\n\n```bash\ngit clone https://github.com/flashinfer-ai/flashinfer.git --recursive\ncd flashinfer\npython -m pip install -v .\n```\n\n**For development**, install in editable mode:\n\n```bash\npython -m pip install --no-build-isolation -e . -v\n```\n\n> **Note:** When using `--no-build-isolation`, pip does not automatically install build dependencies. FlashInfer requires `setuptools>=77`. If you encounter an error like `AttributeError: module 'setuptools.build_meta' has no attribute 'prepare_metadata_for_build_editable'`, upgrade pip and setuptools first:\n> ```bash\n> python -m pip install --upgrade pip setuptools\n> ```\n\nBuild optional packages:\n\n```bash\n\n---\n\n# Verify installation and view configuration\nflashinfer show-config","github_created_at":"2023-07-22T00:29:04+00:00","created_at":"2026-07-11T11:42:32.14183+00:00","updated_at":"2026-09-20T06:32:39.205861+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"attention","name":"attention"},{"slug":"cuda","name":"cuda"},{"slug":"distributed-inference","name":"distributed-inference"},{"slug":"gpu","name":"gpu"},{"slug":"jit","name":"jit"},{"slug":"large-large-models","name":"large-large-models"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"moe","name":"moe"}],"trust":{"provenance":{"is_fork":false,"github_id":669331976,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-19T18:29:29.862Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":30,"days_since_push":0,"last_release_at":"2026-09-16T10:06:25Z","stars_delta_30d":428,"open_issues_delta_30d":124},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:42:33.236Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-19T18:29:30.899Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-19T18:29:30.899Z"},"languages":{"value":["cuda","python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-19T18:29:30.899Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-19T18:29:30.899Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":[],"requires_docker":false},"constraints":{"requires_docker":false},"when_to_use":["When aiming to deploy large language models efficiently using CUDA capabilities, maximizing GPU utilization with FlashInfer can be advantageous.","For projects requiring distributed inference across GPUs, FlashInfer's focus on integrating with PyTorch and leveraging attention mechanisms might provide a competitive edge."],"when_not_to_use":["If the project does not involve large-scale language models or has limited GPU resources, FlashInfer’s specialized features may offer fewer benefits.","For those preferring frameworks integrated closely with other deep learning APIs beyond PyTorch, considering alternatives might better align with diverse tooling requirements."],"source":"enrich:decision_facts","observed_at":"2026-07-17T13:28:12.038Z"},"constraint_facets":{"requires_docker":false},"decision_summary":[{"label":"Adopt for","value":"FlashInfer is a Python library that optimizes inference for large-scale language models through the application of CUDA and GPU support."},{"label":"License detail","value":"Apache-2.0"}]}}