--- title: "flash-linear-attention" type: "tool" slug: "fla-org-flash-linear-attention" canonical_url: "https://www.graphcanon.com/tools/fla-org-flash-linear-attention" github_url: "https://github.com/fla-org/flash-linear-attention" homepage_url: "https://github.com/fla-org/flash-linear-attention" stars: 5308 forks: 571 primary_language: "Python" license: "MIT" categories: ["llm-frameworks"] tags: ["large-language-models", "natural-language-processing", "machine-learning-systems", "sequence-modeling"] updated_at: "2026-07-07T18:40:10.610387+00:00" --- # flash-linear-attention > Efficient implementations for emerging model architectures Flash Linear Attention provides hardware-efficient building blocks, training-ready layers, and components for modern sequence models. It supports a variety of attention mechanisms and hybrid LLM architectures. ## Facts - Repository: https://github.com/fla-org/flash-linear-attention - Homepage: https://github.com/fla-org/flash-linear-attention - Stars: 5,308 · Forks: 571 · Open issues: 68 · Watchers: 34 - Primary language: Python - License: MIT - Last pushed: 2026-07-07T17:26:25+00:00 ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) ## Tags large language models, natural-language-processing, machine-learning-systems, sequence-modeling ## Related tools - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659) - [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019) - [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575) - [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702) - [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711) - [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,610) - [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,150) ## README (excerpt) ```text
Flash Linear Attention

💥 Flash Linear Attention brings together hardware-efficient building blocks, training-ready layers, and components for modern sequence models, spanning linear attention, sparse attention, state space models, and hybrid LLM architectures. All implementations are platform-agnostic and verified on NVIDIA, AMD, and Intel hardware. Pull requests are welcome!

-------- * [News](#news) * [Models](#models) * [Installation](#installation) * [Usage](#usage) * [Token Mixing](#token-mixing) * [Fused Modules](#fused-modules) * [Generation](#generation) * [Hybrid Models](#hybrid-models) * [Training](#training) * [Evaluation](#evaluation) * [Benchmarks](#benchmarks) * [Citation](#citation) * [Star History](#star-history) * [Acknowledgements](#acknowledgements) ## News - [2026-07] 🚀 Add [FlashQLA](https://github.com/QwenLM/FlashQLA) backend for [Gated DeltaNet](fla/ops/gated_delta_rule). - [2026-06] 🔭 Add Parallax implementation to `fla` ([paper](https://arxiv.org/abs/2605.29157)). - [2026-06] 🧱 Add Wall attention implementation to `fla` ([blog](https://blog.tilderesearch.com/blog/wall-attn)). - [2026-05] 🚪 Add Gated DeltaNet 2 (GDN-2) implementation to `fla` ([paper](https://arxiv.org/abs/2605.22791)). - [2026-05] 🦅 Add Raven implementation to `fla` ([repo](https://github.com/goombalab/raven)). - [2026-05] 🚀 Add [YOCO](https://arxiv.org/abs/2405.05254) (You Only Cache Once) implementation to `fla`. - [2026-05] ⚡ Add fused [AttnRes](fla/ops/attnres) support to `fla` ([paper](https://arxiv.org/abs/2603.15031)). - [2026-04] 🐍 Add Mamba3 implementation to `fla` ([paper](https://arxiv.org/abs/2603.15569)). - [2026-04] 🧱 Add [MoBA](https://arxiv.org/abs/2502.13189) (Mixture of Block Attention) implementation to `fla`, with [FlashMoBA](https://github.com/mit-han-lab/flash-moba) backend support. - [2026-04] 🧱 Add [TileLang](https://github.com/tile-ai/tilelang) backend support for selected kernels. - [2026-04] 🎯 Add [GPT-OSS](https://openai.com/index/introducing-gpt-oss/)-style attention sink support to `fla`'s attention kernels. - [2026-03] 🚀 Add [Context Parallel](fla/ops/cp/README.md) support for KDA and GDN, enabling efficient distributed training across sequence dimension. - [2025-10] 🌘 Add Kimi Delta Attention (KDA) implementation to `fla` ([paper](https://arxiv.org/abs/2510.26692)). - [2025-09] 🌲 Add DeltaFormer implementation to `fla` ([paper](https://arxiv.org/abs/2505.19488v1)). - [2025-09] 🐻 Thrilled to announce that [GDN](fla/ops/gated_delta_rule) has been integrated into Qwen3-Next. Check out their [blog post](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list) for more info! - [2025-08] 🌲 Add Log-Linear Attention implementation to `fla` ([paper](https://arxiv.org/abs/2506.04761)). - [2025-08] 🎓 Add MoM implementation to `fla` ([paper](https://arxiv.org/abs/2502.13685)).
Older news - [2025-07] 🐳 Add MLA implementation to `fla` ([paper](https://arxiv.org/abs/2405.04434)). - [2025-07] 🛣️ Add PaTH Attention implementation to `fla` ([paper](https://arxiv.org/abs/2505.16381)). - [2025-06] 🎉 Add MesaNet implementation to `fla` ([paper](https://arxiv.org/abs/2506.05233)). - [2025-06] 🐍 Add Comba implementation to `fla` ([paper](https://arxiv.org/abs/2506.02475)). - [2025-05] 🎉 Add Rodimus* implementation to `fla` ([paper](https://arxiv.org/abs/2410.06577)). - [2025-04] 🎉 Add DeltaProduct implementation to `fla` ([paper](https://arxiv.org/abs/2502.10297)). - [2025-04] 🎉 Add FoX implementation to `fla` ([paper](https://arxiv.org/abs/2503.02130)). - [2025-03] ~~We have changed the default `initializer_range` to the magic 🐳 0.006~~ The `initializer_range` was rolled back to the default value of 0.02. For actual training, we recommend trying both. - [2025-02] 🐳 Add NSA implementations to `fla`. See kernels [he ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/fla-org-flash-linear-attention`](/api/graphcanon/tools/fla-org-flash-linear-attention) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_