{"data":{"slug":"nvidia-star-attention","name":"Star-Attention","tagline":"Efficient LLM Inference over Long Sequences","github_url":"https://github.com/NVIDIA/Star-Attention","owner":"NVIDIA","repo":"Star-Attention","owner_avatar_url":"https://avatars.githubusercontent.com/u/1728152?v=4","primary_language":"Python","stars":392,"forks":25,"topics":["attention-mechanism","large-language-models","llm-inference"],"archived":false,"github_pushed_at":"2025-06-25T19:36:21+00:00","maintenance_label":"Dormant","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/nvidia-star-attention","markdown_url":"https://www.graphcanon.com/tools/nvidia-star-attention.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nvidia-star-attention","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nvidia-star-attention","description":"Efficient LLM Inference over Long Sequences","homepage_url":"https://arxiv.org/abs/2411.17116","license":"Apache-2.0","open_issues":1,"watchers":8,"ai_summary":"A Python project for efficient inference of large language models with long sequences using star-attention mechanisms.","readme_excerpt":"# Star Attention: Efficient LLM Inference over Long Sequences\n\nThis repository contains code for the paper [Star Attention: Efficient LLM Inference over Long Sequences](https://arxiv.org/abs/2411.17116). Star Attention is a novel block-sparse attention mechanism designed to enable efficient inference on long sequences in transformer-based LLMs. The method operates in two phases:\n1. **Phase 1 - Context Encoding**: The context tokens are processed using blockwise-local attention, with the context segmented into blocks where each block is prefixed with an anchor block.\n2. **Phase 2 - Query Processing and Token Generation**: The query and response tokens attend to all prior cached tokens through sequence-global attention.\n\nStar Attention **improves the inference time by up to 11x** while **preserving 97-100% of accuracy**. The method is **compatible with most Transformer-based LLMs trained with global attention, operating seamlessly out-of-the-box without additional training/finetuning.** Furthermore, Star Attention is **orthogonal to other optimization methods**, including Flash Attention and KV cache compression techniques, allowing for potential combined enhancements.\n\nThis codebase contains the implementation of Star Attention in PyTorch using the [HuggingFace Transformers](https://github.com/huggingface/transformers) library, along with the code for launching inference with Star Attention on two benchmarks: RULER and BABILong.\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n ","github_created_at":"2024-11-19T19:10:52+00:00","created_at":"2026-07-11T11:46:47.514814+00:00","updated_at":"2026-08-25T12:01:28.853361+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"attention-mechanism","name":"attention-mechanism"},{"slug":"large-language-models","name":"large language models"},{"slug":"llm-inference","name":"llm-inference"}],"trust":{"provenance":{"is_fork":false,"github_id":891123403,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-25T12:01:28.032Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":425,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:46:48.953Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-25T12:01:28.510Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-25T12:01:28.510Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-25T12:01:28.510Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["For applications requiring handling very large input sequences","When you need to enhance computational efficiency with star-attention mechanism"],"when_not_to_use":["If your use case involves short sequence processing only","In scenarios where traditional attention mechanisms yield adequate results without performance loss"],"source":"enrich:decision_facts","observed_at":"2026-07-15T10:16:22.597Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Star-Attention specializes in long sequence inference of large language models using star-attention to maintain efficiency."}]}}
ModelSeq. Len.
(K)		Block Size
(K)		Ring-Attn
Acc. (%)		Star-Attn
Δ Acc.Δ Speedup
meta-llama
Llama3.1-8B-Instruct		16492.22-0.94%1.1x
32887.53+1.17%1.2x
641684.79-1.42%1.8x
1283276.31-1.90%2.7x
meta-llama
Llama-3.1-70B-Instruct		16495.09-2.71%1.7x
32894.61-2.55%