{"data":{"slug":"fasterdecoding-medusa","name":"Medusa","tagline":"Framework for accelerating LLM generation using multiple decoding heads","github_url":"https://github.com/FasterDecoding/Medusa","owner":"FasterDecoding","repo":"Medusa","owner_avatar_url":"https://avatars.githubusercontent.com/u/144572371?v=4","primary_language":"Jupyter Notebook","stars":2767,"forks":205,"topics":["llm","llm-inference"],"archived":false,"github_pushed_at":"2024-06-25T12:23:04+00:00","maintenance_label":"Dormant","stars_delta_30d":9,"url":"https://www.graphcanon.com/tools/fasterdecoding-medusa","markdown_url":"https://www.graphcanon.com/tools/fasterdecoding-medusa.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/fasterdecoding-medusa","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=fasterdecoding-medusa","description":"Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads","homepage_url":"https://sites.google.com/view/medusa-llm","license":"Apache-2.0","open_issues":57,"watchers":29,"ai_summary":"Medusa is designed to speed up the inference process of large language models through parallelized decoding strategies.","readme_excerpt":"<img src=\"assets/logo.png\" alt=\"Medusa\" width=\"100\" align=\"left\"><div align=\"center\"><h1>&nbsp;Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads</h1></div>\n\n<p align=\"center\">\n| <a href=\"https://sites.google.com/view/\nmedusa-llm\"><b>Blog</b></a> | <a href=\"https://arxiv.org/abs/2401.10774\"><b>Report</b></a> | <a href=\"ROADMAP.md\"><b>Roadmap</b></a> |\n</p>\n\n---\n*News* 🔥\n- [2024/1] Medusa technical report is now available on [arXiv](https://arxiv.org/abs/2401.10774). We've added multiple new features, including Medusa-2 recipe for full-model training, self-distillation for adding Medusa to any fine-tuned LLM, etc. The new results show a 2.2-3.6x speedup over the original model on a range of LLMs.\n\n---\n## Introduction\n\nMedusa is a simple framework that democratizes the acceleration techniques for LLM generation with multiple decoding heads.\n\n<div align=\"center\">\n  <picture>\n  <img src=\"assets/medusa_demo.gif\" width=\"80%\">\n  </picture>\n  <br>\n  <div align=\"center\" width=\"80%\">\n  <em>Medusa-1 on Vicuna-7b.</em>\n  </div>\n  <br>\n</div>\n\n\nWe aim to tackle the three pain points of popular acceleration techniques like speculative decoding:\n\n- Requirement of a good draft model.\n- System complexity.\n- Inefficiency when using sampling-based generation.\n\n\n<div align=\"center\">\n  <picture>\n  <img src=\"assets/medusa_pipeline.jpg\" width=\"60%\">\n  </picture>\n  <br>\n  <div align=\"left\" width=\"80%\">\n  <em>Medusa adds extra \"heads\" to LLMs to predict multiple future tokens simultaneously. When augmenting a model with Medusa, the original model stays untouched, and only the new heads are fine-tuned during training. During generation, these heads each produce multiple likely words for the corresponding position. These options are then combined and processed using a tree-based attention mechanism. Finally, a typical acceptance scheme is employed to pick the longest plausible prefix from the candidates for further decoding.</em>\n  </div>\n  <br>\n</div>\n\nWe aim to solve the challenges associated with speculative decoding by implementing the following ideas:\n\n- Instead of introducing a new model, we train multiple decoding heads on the *same* model.\n- The training is parameter-efficient so that even the \"GPU-Poor\" can do it. And since there is no additional model, there is no need to adjust the distributed computing setup.\n- Relaxing the requirement of matching the distribution of the original model makes the non-greedy generation even faster than greedy decoding.\n\nIn the initial release, our primary focus is on optimizing Medusa for a batch size of 1—a setting commonly utilized for local model hosting. In this configuration, Medusa delivers approximately a 2x speed increase across a range of Vicuna models. We are actively working to extend Medusa's capabilities by integrating it into additional inference frameworks, with the aim of achieving even greater performance gains and extending Medusa to broader settings.\n\n<p align=\"center\">\n  <picture>\n  <img src=\"assets/medusa_speedup_cmp.jpg\" width=\"45%\">\n  </picture>\n</p>\n\nIn the updated version, we add support for full-model training, called Medusa-2 (compared to Medusa-1, which only trains the new heads), which requires a special recipe that adds the speculative prediction ability while keeping the original model's performance.\n\nWe also add support for self-distillation, which allows us to add Medusa to any fine-tuned LLM without requiring the availability of the original training data.\n\n## Contents\n- [Introduction](#introduction)\n- [Contents](#contents)\n- [Installation](#installation)\n  - [Method 1: With pip (may not be the latest version)](#method-1-with-pip-may-not-be-the-latest-version)\n  - [Method 2: From the source (recommended)](#method-2-from-the-source-recommended)\n  - [Model Weights](#model-weights)\n  - [Inference](#inference)\n  - [Training](#training)\n  - [Training (legacy)](#training-legacy)\n  - [Push to Hugging Face Hub](#push-to-hugging-face-hub)\n- [Citation","github_created_at":"2023-09-10T06:14:07+00:00","created_at":"2026-07-11T11:43:08.625223+00:00","updated_at":"2026-08-24T18:01:40.385656+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"acceleration","name":"acceleration"},{"slug":"decoding","name":"decoding"},{"slug":"inference","name":"inference"},{"slug":"llm","name":"llm"}],"trust":{"provenance":{"is_fork":false,"github_id":689541424,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-24T18:01:39.554Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":790,"last_release_at":"2023-09-11T20:35:12Z","stars_delta_30d":9,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:43:09.964Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-24T18:01:40.092Z"},"languages":{"value":["jupyter notebook","python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-24T18:01:40.092Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-24T18:01:40.092Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to accelerate inference times for large language models without compromising on output quality.","For applications requiring real-time interaction where rapid response times are critical and multiple decoding paths can be utilized."],"when_not_to_use":["If your model does not benefit from parallelized decoding, such as when the model architecture inherently limits parallel execution efficiency.","In scenarios where the computational resources required for multiple decoding heads exceed what is available or cost-effective within your infrastructure."],"source":"enrich:decision_facts","observed_at":"2026-07-14T19:49:22.684Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Medusa enables quicker language model inference with parallel decoding strategies."}]}}