MInference
Accelerates Long-context LLMs' inference through approximate sparse calculation for attention.
GraphCanon updated 2w · GitHub synced 2w · 28 views this month
Decision brief
MInference accelerates long-context LLMs' inference by up to 10x via approximate sparse calculation techniques while preserving model accuracy.
Good fit when
- MInference is ideal for scenarios where significant reduction in inference latency is needed without sacrificing the accuracy of long-context LLM outputs.
- MInference should be used when developers are working with models that support Transformers >= 4.46.0 and require fast pre-filling speeds to meet real-time performance requirements.
Avoid when
- Avoid using MInference if your application does not benefit from or cannot tolerate slight variations in inference times due to its use of approximate sparse calculation.
- MInference might not be suitable for applications where the model's accuracy is critical and any reduction in the precision introduced by approximations would be detrimental.
- Requirements:
- Min 8 GB RAM; MInference requires at least Torch and optionally FlashAttention-2 for maximum efficiency.; Triton for faster deployment and integration.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (120d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for Microsoft. Display-only - separate from trust and ranking.
- Company
- Microsoft·GitHub org profile·1mo
- Employees
- 221,000·Wikidata (P1128 employees)·1mo
- Commercial model
- Pure OSS·GitHub org profile (public repos)·1mo
Install
pip install MInferenceSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
MInference is a Python library that aims to reduce the latency of long-context Large Language Models (LLMs) by up to 10x during pre-filling, while maintaining accuracy. It leverages frameworks like Torch and FlashAttention-2 (optional), and is compatible with Transformers >= 4.46.0.
Capability facts
- Languages
- python
Source: github.language · Aug 7, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Tags
README
Requirements
- Torch
- FlashAttention-2 (Optional)
- Triton
- Transformers >= 4.46.0
To get started with MInference, simply install it using pip:
pip install minference
For agents
This page has a .md twin and JSON over the API.