GraphCanon

flash-linear-attention

fla-org/flash-linear-attention

Efficient implementations for emerging model architectures

GitHub Website
5.3k
Stars
571
Forks
68
Open issues
34
Watchers
Python MITLast pushed Jul 7, 2026

Overview

Flash Linear Attention provides hardware-efficient building blocks, training-ready layers, and components for modern sequence models. It supports a variety of attention mechanisms and hybrid LLM architectures.

Categories

LLM Frameworks

Tags

large language modelsnatural-language-processingmachine-learning-systemssequence-modeling

Similar tools

ollama

ollama/ollama

Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

176k 17kGo

prompts.chat

f/prompts.chat

The world's largest open-source prompt library for AI

165k 21kHTML

transformers

huggingface/transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models

162k 34kPython

open-webui

open-webui/open-webui

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

145k 21kPython

awesome-llm-apps

Shubhamsaboo/awesome-llm-apps

100+ AI Agent & RAG apps you can actually run — clone, customize, ship.

117k 17kPython

LLMs-from-scratch

rasbt/LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch

99k 15kJupyter Notebook

Install

pip install flash-linear-attention
PyPI Docs / website

README


💥 Flash Linear Attention brings together hardware-efficient building blocks, training-ready layers, and components for modern sequence models, spanning linear attention, sparse attention, state space models, and hybrid LLM architectures. All implementations are platform-agnostic and verified on NVIDIA, AMD, and Intel hardware. Pull requests are welcome!

  • News
  • Models
  • Installation
  • Usage
    • Token Mixing
    • Fused Modules
    • Generation
    • Hybrid Models
  • Training
  • Evaluation
  • Benchmarks
  • Citation
  • Star History
  • Acknowledgements

News

  • [2026-07] 🚀 Add FlashQLA backend for Gated DeltaNet.
  • [2026-06] 🔭 Add Parallax implementation to fla (paper).
  • [2026-06] 🧱 Add Wall attention implementation to fla (blog).
  • [2026-05] 🚪 Add Gated DeltaNet 2 (GDN-2) implementation to fla (paper).
  • [2026-05] 🦅 Add Raven implementation to fla (repo).
  • [2026-05] 🚀 Add YOCO (You Only Cache Once) implementation to fla.
  • [2026-05] ⚡ Add fused AttnRes support to fla (paper).
  • [2026-04] 🐍 Add Mamba3 implementation to fla (paper).
  • [2026-04] 🧱 Add MoBA (Mixture of Block Attention) implementation to fla, with FlashMoBA backend support.
  • [2026-04] 🧱 Add TileLang backend support for selected kernels.
  • [2026-04] 🎯 Add GPT-OSS-style attention sink support to fla's attention kernels.
  • [2026-03] 🚀 Add Context Parallel support for KDA and GDN, enabling efficient distributed training across sequence dimension.
  • [2025-10] 🌘 Add Kimi Delta Attention (KDA) implementation to fla (paper).
  • [2025-09] 🌲 Add DeltaFormer implementation to fla (paper).
  • [2025-09] 🐻 Thrilled to announce that GDN has been integrated into Qwen3-Next. Check out their blog post for more info!
  • [2025-08] 🌲 Add Log-Linear Attention implementation to fla (paper).
  • [2025-08] 🎓 Add MoM implementation to fla (paper).
Older news
  • [2025-07] 🐳 Add MLA implementation to fla (paper).
  • [2025-07] 🛣️ Add PaTH Attention implementation to fla (paper).
  • [2025-06] 🎉 Add MesaNet implementation to fla (paper).
  • [2025-06] 🐍 Add Comba implementation to fla (paper).
  • [2025-05] 🎉 Add Rodimus* implementation to fla (paper).
  • [2025-04] 🎉 Add DeltaProduct implementation to fla (paper).
  • [2025-04] 🎉 Add FoX implementation to fla (paper).
  • [2025-03] We have changed the default initializer_range to the magic 🐳 0.006 The initializer_range was rolled back to the default value of 0.02. For actual training, we recommend trying both.
  • [2025-02] 🐳 Add NSA implementations to fla. See kernels [he