{"data":{"slug":"microsoft-minference","name":"MInference","tagline":"Accelerates Long-context LLMs' inference through approximate sparse calculation for attention.","github_url":"https://github.com/microsoft/MInference","owner":"microsoft","repo":"MInference","owner_avatar_url":"https://avatars.githubusercontent.com/u/6154722?v=4","primary_language":"Python","stars":1225,"forks":80,"topics":[],"archived":false,"github_pushed_at":"2026-04-08T08:04:38+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/microsoft-minference","markdown_url":"https://www.graphcanon.com/tools/microsoft-minference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/microsoft-minference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=microsoft-minference","description":"[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.","homepage_url":"https://aka.ms/MInference","license":"MIT","open_issues":93,"watchers":9,"ai_summary":"MInference is a Python library that aims to reduce the latency of long-context Large Language Models (LLMs) by up to 10x during pre-filling, while maintaining accuracy. It leverages frameworks like Torch and FlashAttention-2 (optional), and is compatible with Transformers >= 4.46.0.","readme_excerpt":"### Requirements\n\n- Torch\n- FlashAttention-2 (Optional)\n- Triton\n- **Transformers >= 4.46.0**\n\nTo get started with MInference, simply install it using pip:\n\n```bash\npip install minference\n```","github_created_at":"2024-05-22T12:59:47+00:00","created_at":"2026-07-11T10:37:05.922556+00:00","updated_at":"2026-08-07T06:01:14.377537+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"attention-mechanism","name":"attention-mechanism"},{"slug":"flashattention-2","name":"flashattention-2"},{"slug":"inference-acceleration","name":"inference acceleration"},{"slug":"long-context-llms","name":"long-context llms"},{"slug":"sparse-calculation","name":"sparse calculation"},{"slug":"torch","name":"torch"}],"trust":{"provenance":{"is_fork":false,"github_id":804362023,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-07T06:01:13.517Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":120,"last_release_at":"2025-06-17T09:29:24Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:37:07.395Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-07T06:01:14.021Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-07T06:01:14.021Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-07T06:01:14.021Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["MInference requires at least Torch and optionally FlashAttention-2 for maximum efficiency.","Triton for faster deployment and integration."],"min_ram_gb":8},"constraints":{"min_ram_gb":8},"when_to_use":["MInference is ideal for scenarios where significant reduction in inference latency is needed without sacrificing the accuracy of long-context LLM outputs.","MInference should be used when developers are working with models that support Transformers >= 4.46.0 and require fast pre-filling speeds to meet real-time performance requirements."],"when_not_to_use":["Avoid using MInference if your application does not benefit from or cannot tolerate slight variations in inference times due to its use of approximate sparse calculation.","MInference might not be suitable for applications where the model's accuracy is critical and any reduction in the precision introduced by approximations would be detrimental."],"source":"enrich:decision_facts","observed_at":"2026-07-12T09:11:55.551Z"},"constraint_facets":{"min_ram_gb":8},"decision_summary":[{"label":"Requirements","value":"Min 8 GB RAM; MInference requires at least Torch and optionally FlashAttention-2 for maximum efficiency.; Triton for faster deployment and integration."},{"label":"Adopt for","value":"MInference accelerates long-context LLMs' inference by up to 10x via approximate sparse calculation techniques while preserving model accuracy."}]}}