{"data":{"slug":"damo-nlp-sg-video-llama","name":"Video-LLaMA","tagline":"Instruction-tuned Audio-Visual Language Model for Video Understanding","github_url":"https://github.com/DAMO-NLP-SG/Video-LLaMA","owner":"DAMO-NLP-SG","repo":"Video-LLaMA","owner_avatar_url":"https://avatars.githubusercontent.com/u/130957594?v=4","primary_language":"Python","stars":3141,"forks":287,"topics":["blip2","cross-modal-pretraining","large-language-models","llama","minigpt4","multi-modal-chatgpt","video-language-pretraining","vision-language-pretraining"],"archived":false,"github_pushed_at":"2024-06-04T07:06:41+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/damo-nlp-sg-video-llama","markdown_url":"https://www.graphcanon.com/tools/damo-nlp-sg-video-llama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/damo-nlp-sg-video-llama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=damo-nlp-sg-video-llama","description":"[EMNLP 2023 Demo] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding","homepage_url":null,"license":"BSD-3-Clause","open_issues":69,"watchers":31,"ai_summary":"This repository focuses on enhancing large language models with capabilities to understand video and audio content.","readme_excerpt":"

\n\"Video-LLaMA\"\n

\n\n\n\n# Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding\n\n\nThis is the repo for the Video-LLaMA project, which is working on empowering large language models with video and audio understanding capabilities. \n\n
\n\n\n\n \n\n
\n\n## News\n-

[2024.06.03] 🚀🚀 We officially launch VideoLLaMA2 with stronger performances and easier-to-use codebase, come to try it out!

\n- [11.14] ⭐️ The current README file is for **Video-LLaMA-2** (LLaMA-2-Chat as language decoder) only, instructions for using the previous version of Video-LLaMA (Vicuna as language decoder) can be found at [here](https://github.com/DAMO-NLP-SG/Video-LLaMA/blob/main/README_Vicuna.md).\n- [08.03] 🚀🚀 Release **Video-LLaMA-2** with [Llama-2-7B/13B-Chat](https://huggingface.co/meta-llama) as language decoder\n - **NO** delta weights and separate Q-former weights anymore, full weights to run Video-LLaMA are all here :point_right: [[7B](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned)][[13B](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-13B-Finetuned)] \n - Allow further customization starting from our pre-trained checkpoints [[7B-Pretrained](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Pretrained)] [[13B-Pretrained](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-13B-Pretrained)]\n- [06.14] **NOTE**: The current online interactive demo is primarily for English chatting and it may **NOT** be a good option to ask Chinese questions since Vicuna/LLaMA does not represent Chinese texts very well. \n- [06.13] **NOTE**: The audio support is **ONLY** for Vicuna-7B by now although we have several VL checkpoints available for other decoders.\n- [06.10] **NOTE**: We have NOT updated the HF demo yet because the whole framework (with the audio branch) cannot run normally on A10-24G. The current running demo is still the previous version of Video-LLaMA. We will fix this issue soon.\n- [06.08] 🚀🚀 Release the checkpoints of the audio-supported Video-LLaMA. Documentation and example outputs are also updated. \n- [05.22] 🚀🚀 Interactive demo online, try our Video-LLaMA (with **Vicuna-7B** as language decoder) at [Hugging Face](https://huggingface.co/spaces/DAMO-NLP-SG/Video-LLaMA) and [ModelScope](https://pre.modelscope.cn/studios/damo/video-llama/summary)!!\n- [05.22] ⭐️ Release **Video-LLaMA v2** built with Vicuna-7B\n- [05.18] 🚀🚀 Support video-grounded chat in Chinese \n - [**Video-LLaMA-BiLLA**](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-Series/resolve/main/finetune-billa7b-zh.pth): we introduce [BiLLa-7B-SFT](https://huggingface.co/Neutralzz/BiLLa-7B-SFT) as language decoder and fine-tune the video-language aligned model (i.e., stage 1 model) with machine-translated [VideoChat](https://github.com/OpenGVLab/InternVideo/tree/main/Data/instruction_data) instructions. \n - [**Video-LLaMA-Ziya**](https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-Series/resolve/main/finetune-ziya13b-zh.pth): same with Video-LLaMA-BiLLA but the language deco","github_created_at":"2023-05-06T15:35:19+00:00","created_at":"2026-07-07T17:36:00.957658+00:00","updated_at":"2026-08-18T00:02:13.095343+00:00","categories":[{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"blip2","name":"blip2"},{"slug":"cross-modal-pretraining","name":"cross-modal-pretraining"},{"slug":"large-language-models","name":"large language models"},{"slug":"llama","name":"llama"},{"slug":"minigpt4","name":"minigpt4"},{"slug":"multi-modal-chatgpt","name":"multi-modal-chatgpt"},{"slug":"video-language-pretraining","name":"video-language-pretraining"},{"slug":"vision-language-pretraining","name":"vision-language-pretraining"}],"trust":{"provenance":{"is_fork":false,"github_id":637126931,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-18T00:02:12.306Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":804,"last_release_at":null,"stars_delta_30d":2,"open_issues_delta_30d":-1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:06:45.144Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-18T00:02:12.806Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-18T00:02:12.806Z"},"license_spdx":{"value":"BSD-3-Clause","source":"github.license","observed_at":"2026-08-18T00:02:12.806Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Ensure access to compatible hardware for video and audio processing tasks.","Consider the availability of Chinese text representation as a potential advantage or limitation based on your project needs."]},"constraints":null,"when_to_use":["When you need to process video content with instruction-tuned multimodal capabilities, especially when working with videos that require both visual and auditory analysis.","For applications requiring a deep understanding of video content in Chinese, as Video-LLaMA includes support for video-grounded chat in this language.","If you are looking to build upon pre-trained checkpoints for customization or fine-tuning towards specific tasks involving video comprehension."],"when_not_to_use":["Do not use when the primary focus is on languages other than English and Chinese, as the model's representation capabilities outside these languages might be limited.","Avoid using Video-LLaMA if you require real-time audio processing in a deployment environment that does not support Vicuna-7B audio branch currently running on A10-24G GPUs."],"source":"enrich:decision_facts","observed_at":"2026-07-14T21:39:34.812Z"},"constraint_facets":null,"decision_summary":[{"label":"Requirements","value":"Ensure access to compatible hardware for video and audio processing tasks.; Consider the availability of Chinese text representation as a potential advantage or limitation based on your project needs."},{"label":"Adopt for","value":"Video-LLaMA is an audio-visual language model that enhances video and audio understanding capabilities for language models."},{"label":"License detail","value":"BSD-3-Clause"}]}}