{"data":{"slug":"ictnlp-llama-omni","name":"LLaMA-Omni","tagline":"End-to-end speech interaction model based on Llama-3.1-8B-Instruct","github_url":"https://github.com/ictnlp/LLaMA-Omni","owner":"ictnlp","repo":"LLaMA-Omni","owner_avatar_url":"https://avatars.githubusercontent.com/u/45630465?v=4","primary_language":"Python","stars":3146,"forks":224,"topics":["large-language-models","multimodal-large-language-models","speech-interaction","speech-language-model","speech-to-speech","speech-to-text"],"archived":false,"github_pushed_at":"2025-05-19T02:24:42+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/ictnlp-llama-omni","markdown_url":"https://www.graphcanon.com/tools/ictnlp-llama-omni.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ictnlp-llama-omni","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ictnlp-llama-omni","description":"LLaMA-Omni is a low-latency and high-quality end-to-end speech interaction model built upon Llama-3.1-8B-Instruct, aiming to achieve speech capabilities at the GPT-4o level.","homepage_url":"https://arxiv.org/abs/2409.06666","license":"Apache-2.0","open_issues":52,"watchers":34,"ai_summary":"LLaMA-Omni is designed for high-quality and low-latency speech interaction capabilities, aiming to achieve GPT-4o levelspeech functionalities.","readme_excerpt":"## Install\n\n1. Clone this repository.\n\n```shell\ngit clone https://github.com/ictnlp/LLaMA-Omni\ncd LLaMA-Omni\n```\n\n2. Install packages.\n\n```shell\nconda create -n llama-omni python=3.10\nconda activate llama-omni\npip install pip==24.0\npip install -e .\n```\n\n3. Install `fairseq`.\n\n```shell\ngit clone https://github.com/pytorch/fairseq\ncd fairseq\npip install -e . --no-build-isolation\n```\n\n4. Install `flash-attention`.\n\n```shell\npip install flash-attn --no-build-isolation\n```\n\n---\n\n## Quick Start\n\n1. Download the `Llama-3.1-8B-Omni` model from 🤗[Huggingface](https://huggingface.co/ICTNLP/Llama-3.1-8B-Omni). \n\n2. Download the `Whisper-large-v3` model.\n\n```shell\nimport whisper\nmodel = whisper.load_model(\"large-v3\", download_root=\"models/speech_encoder/\")\n```\n\n3. Download the unit-based HiFi-GAN vocoder.\n\n```shell\nwget https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj/g_00500000 -P vocoder/\nwget https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj/config.json -P vocoder/\n```\n\n---\n\n## LICENSE\n\nOur code is released under the Apache-2.0 License. Our model is intended for academic research purposes only and may **NOT** be used for commercial purposes.\n\nYou are free to use, modify, and distribute this model in academic settings, provided that the following conditions are met:\n\n- **Non-commercial use**: The model may not be used for any commercial purposes.\n- **Citation**: If you use this model in your research, please cite the original work.","github_created_at":"2024-09-10T12:21:53+00:00","created_at":"2026-07-07T17:36:02.501083+00:00","updated_at":"2026-07-30T12:00:30.793024+00:00","categories":[{"slug":"speech-audio","name":"Speech & Audio","url":"https://www.graphcanon.com/categories/speech-audio","markdown_url":"https://www.graphcanon.com/categories/speech-audio.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/speech-audio"}],"tags":[{"slug":"large-language-models","name":"large language models"},{"slug":"multimodal-large-language-models","name":"multimodal-large-language-models"},{"slug":"speech-interaction","name":"speech-interaction"},{"slug":"speech-language-model","name":"speech-language-model"},{"slug":"speech-to-speech","name":"speech-to-speech"},{"slug":"speech-to-text","name":"speech-to-text"}],"trust":{"provenance":{"is_fork":false,"github_id":855162700,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-30T12:00:29.344Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":437,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:16:19.441Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-30T12:00:30.406Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-30T12:00:30.406Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-30T12:00:30.406Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When targeting low-latency, high-quality end-to-end speech interactions that need to be performed in an academic research environment.","- If you are working on projects that require integrating both speech and text modalities for processing or generating responses, especially focusing on tasks where reducing latency is critical."],"when_not_to_use":["- Avoid LLaMA-Omni if your project requires commercial deployment since its usage rights are strictly non-commercial.","- If real-time interaction constraints are less critical than achieving high-quality speech output, another tool with more flexibility regarding latency and deployment options may be preferable."],"source":"enrich:decision_facts","observed_at":"2026-07-10T00:22:42.771Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"LLaMA-Omni is a specialized multimodal large language model tailored for enhancing speech interaction capabilities through advanced integration of speech-to-speech and speech-to-text functionalities."}]}}