| Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack |
🔥 We have built a vLLM website to help you get started with vLLM. Please visit [vllm.ai](https://vllm.ai) to learn more. For events, please visit [vllm.ai/events](https://vllm.ai/events) to join us. --- ## About vLLM is a fast and easy-to-use library for LLM inference and serving. Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html) - Continuous batching of incoming requests, chunked prefill, prefix caching - Fast and flexible model execution with piecewise and full CUDA/HIP graphs - Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html) - Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton - Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL - Speculative decoding including n-gram, suffix, EAGLE, DFlash - Automatic kernel generation and graph-level transformations using torch.compile - Disaggregated prefill, decode, and encode vLLM is flexible and easy to use with: - Seamless integration with popular Hugging Face models - High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more - Tensor, pipeline, data, expert, and context parallelism for distributed inference - Streaming outputs - Generation of structured outputs using xgrammar or guidance - Tool calling and reasoning parsers - OpenAI-compatible API server, plus Anthropic Messages API and gRPC support - Efficient multi-LoRA support for dense and MoE layers - Support for NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more. vLLM seamlessly supports 200+ model architectures on Hugging Face, including: - Decoder-only LLMs (e.g., Llama, Qwen, Gemma) - Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS) - Hybrid attention and state-space models (e.g., Mamba, Qwen3.5) - Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral) - Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT) - Reward and classification models (e.g., Qwen-Math) Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html). ## Getting Started Install vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`: ```bash uv pip install vllm ``` Or [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development. Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more. - [Installation](https://docs.vllm.ai/en/latest/getting_started/installation. ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/vllm-project-vllm`](/api/graphcanon/tools/vllm-project-vllm) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_