🦜 VideoChat Family: Ask-Anything

| | | |
[VideoChat-7B-8Bit] End2End ChatBOT for video and image. [InternVideo2-Chat-8B-HD]

中文 README 及 中文交流群 | Paper

⭐️: We are also working on a updated version, stay tuned!

:fire: Updates

• 2025/01/18: We release videochat-flash and videochat-tpo to extend MLLMs' capabilities on both long and accurate video understanding. videochat-flash sets new records in mutiple video benchmarks (for both short and long videos), improving code usability by leveaging LLaVA and others. videochat-tpo exploits classical vision task annotations (e.g. tracking) to optimize MLLMs in a DPO manner, enhancing MLLMs' performance and enabling capabilities in tracking, segmentation, and more.
• 2024/06/25: We release the branch of videochat2 using vllm, speed up the inference of videochat2.
• 2024/06/19: 🎉🎉 Our VideoChat2 achieves the best performances among the open-sourced VideoLLMs on MLVU, a multi-task long video understanding benchmark.
• 2024/06/13: Fix some bug and give testing scripts/
  • :warning: We replace some repeated (~30) QAs in MVBench, which may only affect the results by 0.5%.
  • :loudspeaker: We give the scripts for testing EgoSchema and Video-MME, please check the demo_mistral.ipynb and demo_mistral_hd.ipynb.
• 2024/06/07: :fire::fire::fire: We release VideoChat2_HD, which is fine-tuned with high-resolution data and is capable of handling more diverse tasks. It showcases better performance on different benchmarks, especially for detailed captioning. Furthermore, it achieves 54.8% on Video-MME, the best score among 7B MLLMs. Have a try! 🏃🏻‍♀️🏃🏻
• 2024/06/06: We release VideoChat2_phi3, a faster model with robust performaces.
• 2024/05/22: We release VideoChat2_mistral, which shows better capacity on diverse tasks (60.4% on MVBench, 78.6% on NExT-QA, 63.8% on STAR, 46.4% on TVQA, 54.4% on EgoSchema-full and 80.5% on IntentQA). More details have been updated in the paper.
• 2024/04/05 MVBench is selected as Poster (Highlight)!
• 2024/2/27 MVBench is accepted by CVPR2024.
• 2023/