Video-LLaMA
DAMO-NLP-SG/Video-LLaMA
Video-LLaMA: Instruction-tuned Audio-Visual Language Model
Overview
An instruction-tuned model for understanding videos with audio and visual components, based on LLaMA. This repository hosts Video-LLaMA, a project aimed at enabling large language models to understand video and audio content.
Categories
Tags
Similar tools
ollama
ollama/ollama
Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
prompts.chat
f/prompts.chat
The world's largest open-source prompt library for AI
transformers
huggingface/transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models
langflow
langflow-ai/langflow
Langflow is a powerful platform for building and deploying AI-powered agents and workflows.
firecrawl
firecrawl/firecrawl
The API to search, scrape, and interact with the web at scale.
open-webui
open-webui/open-webui
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
Install
pip install Video-LLaMAREADME
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
This is the repo for the Video-LLaMA project, which is working on empowering large language models with video and audio understanding capabilities.
News
-
[2024.06.03] 🚀🚀 We officially launch VideoLLaMA2 with stronger performances and easier-to-use codebase, come to try it out!
- [11.14] ⭐️ The current README file is for Video-LLaMA-2 (LLaMA-2-Chat as language decoder) only, instructions for using the previous version of Video-LLaMA (Vicuna as language decoder) can be found at here.
- [08.03] 🚀🚀 Release Video-LLaMA-2 with Llama-2-7B/13B-Chat as language decoder
- NO delta weights and separate Q-former weights anymore, full weights to run Video-LLaMA are all here :point_right: [7B][13B]
- Allow further customization starting from our pre-trained checkpoints [7B-Pretrained] [13B-Pretrained]
- [06.14] NOTE: The current online interactive demo is primarily for English chatting and it may NOT be a good option to ask Chinese questions since Vicuna/LLaMA does not represent Chinese texts very well.
- [06.13] NOTE: The audio support is ONLY for Vicuna-7B by now although we have several VL checkpoints available for other decoders.
- [06.10] NOTE: We have NOT updated the HF demo yet because the whole framework (with the audio branch) cannot run normally on A10-24G. The current running demo is still the previous version of Video-LLaMA. We will fix this issue soon.
- [06.08] 🚀🚀 Release the checkpoints of the audio-supported Video-LLaMA. Documentation and example outputs are also updated.
- [05.22] 🚀🚀 Interactive demo online, try our Video-LLaMA (with Vicuna-7B as language decoder) at Hugging Face and ModelScope!!
- [05.22] ⭐️ Release Video-LLaMA v2 built with Vicuna-7B
- [05.18] 🚀🚀 Support video-grounded chat in Chinese
- Video-LLaMA-BiLLA: we introduce BiLLa-7B-SFT as language decoder and fine-tune the video-language aligned model (i.e., stage 1 model) with machine-translated VideoChat instructions.
- Video-LLaMA-Ziya: same with Video-LLaMA-BiLLA but the language deco