{"data":{"slug":"ggml-org-llama-cpp","name":"llama.cpp","tagline":"LLM inference in C/C++","github_url":"https://github.com/ggml-org/llama.cpp","owner":"ggml-org","repo":"llama.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/134263123?v=4","primary_language":"C++","stars":119588,"forks":20314,"topics":["ggml"],"archived":false,"github_pushed_at":"2026-07-07T22:35:25+00:00","url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp","markdown_url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ggml-org-llama-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ggml-org-llama-cpp","description":"LLM inference in C/C++","homepage_url":"https://llama.app","license":"MIT","open_issues":1826,"watchers":773,"ai_summary":"A C/C++ implementation of large language model (LLM) inference.","readme_excerpt":"# llama.cpp\n\n\n\n\n\n\n\n\n\n[Manifesto](https://github.com/ggml-org/llama.cpp/discussions/205) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md)\n\nLLM inference in C/C++\n\n## Recent API changes\n\n- [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289)\n- [Changelog for `llama-server` REST API](https://github.com/ggml-org/llama.cpp/issues/9291)\n\n## Hot topics\n\n- **Hugging Face cache migration: models downloaded with `-hf` are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.**\n- **[guide : using the new WebUI of llama.cpp](https://github.com/ggml-org/llama.cpp/discussions/16938)**\n- [guide : running gpt-oss with llama.cpp](https://github.com/ggml-org/llama.cpp/discussions/15396)\n- [[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗](https://github.com/ggml-org/llama.cpp/discussions/15313)\n- Support for the `gpt-oss` model with native MXFP4 format has been added | [PR](https://github.com/ggml-org/llama.cpp/pull/15091) | [Collaboration with NVIDIA](https://blogs.nvidia.com/blog/rtx-ai-garage-openai-oss) | [Comment](https://github.com/ggml-org/llama.cpp/discussions/15095)\n- Multimodal support arrived in `llama-server`: [#12898](https://github.com/ggml-org/llama.cpp/pull/12898) | [documentation](./docs/multimodal.md)\n- VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode\n- Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim\n- Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669\n- Hugging Face GGUF editor: [discussion](https://github.com/ggml-org/llama.cpp/discussions/9268) | [tool](https://huggingface.co/spaces/CISCai/gguf-editor)\n- WebGPU support is now available in the browser, see a blog/demo introducing it [here](https://reeselevine.github.io/llamas-on-the-web/).\n\n----\n\n## Quick start\n\nGetting started with llama.cpp is straightforward. Here are several ways to install it on your machine:\n\n- Install `llama.cpp` using [brew, nix, winget, or conda-forge](docs/install.md)\n- Run with Docker - see our [Docker documentation](docs/docker.md)\n- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)\n- Build from source by cloning this repository - check out [our build guide](docs/build.md)\n\nOnce installed, you'll need a model to work with. Head to the [Obtaining and quantizing models](#obtaining-and-quantizing-models) section to learn more.\n\nExample command:\n\n```sh\n# Use a local model file\nllama-cli -m my_model.gguf\n\n# Or download and run a model directly from Hugging Face\nllama-cli -hf ggml-org/gemma-3-1b-it-GGUF\n\n# Launch OpenAI-compatible API server\nllama-server -hf ggml-org/gemma-3-1b-it-GGUF\n```\n\n## Description\n\nThe main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide\nrange of hardware - locally and in the cloud.\n\n- Plain C/C++ implementation without any dependencies\n- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks\n- AVX, AVX2, AVX512 and AMX support for x86 architectures\n- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures\n- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use\n- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)\n- Vulkan and SYCL backend support\n- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity\n\nThe `llama.cpp` project is the main playground for developing new features for the [ggml](https://github.com/ggml-org/ggml) library.\n\n<details>\n<summary>Models</summary>\n\nTypically finetunes of the base models below are supported as well.\n\nInstructions for adding support for new models: [HOWTO-add-model.md](docs","github_created_at":"2023-03-10T18:58:00+00:00","created_at":"2026-07-07T22:37:23.741655+00:00","updated_at":"2026-07-07T22:37:29.11343+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ggml","name":"ggml"},{"slug":"llm","name":"llm"},{"slug":"c","name":"c++"},{"slug":"inference","name":"inference"}]}}