--- title: "NExT-GPT" type: "tool" slug: "next-gpt-next-gpt" canonical_url: "https://www.graphcanon.com/tools/next-gpt-next-gpt" github_url: "https://github.com/NExT-GPT/NExT-GPT" homepage_url: "https://next-gpt.github.io/" stars: 3636 forks: 361 primary_language: "Python" license: "BSD-3-Clause" categories: ["llm-frameworks", "model-training"] tags: ["multi-modal-chatgpt", "llm", "instruction-tuning", "large-language-models", "gpt-4", "chatgpt", "foundation-models", "multimodal"] updated_at: "2026-07-07T20:01:15.618558+00:00" --- # NExT-GPT > Code and models for NExT-GPT: Any-to-Any Multimodal Large Language Model Repository for the ICML 2024 paper presenting NExT-GPT, an end-to-end multimodal language model capable of processing inputs and generating outputs in various combinations of text, image, video, audio, etc. ## Facts - Repository: https://github.com/NExT-GPT/NExT-GPT - Homepage: https://next-gpt.github.io/ - Stars: 3,636 · Forks: 361 · Open issues: 81 · Watchers: 59 - Primary language: Python - License: BSD-3-Clause - Last pushed: 2025-05-13T09:57:47+00:00 ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) - [Model Training](/categories/model-training.md) ## Tags multi-modal-chatgpt, llm, instruction-tuning, large-language-models, gpt-4, chatgpt, foundation-models, multimodal ## Related tools - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,664) - [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,025) - [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,350) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,582) - [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702) - [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,715) - [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,619) - [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,189) ## README (excerpt) ```text # NExT-GPT: Any-to-Any Multimodal LLM [Shengqiong Wu](https://chocowu.github.io/), [Hao Fei](http://haofei.vip/)*, [Leigang Qu](#), [Wei Ji](https://jiwei0523.github.io/), and [Tat-Seng Chua](https://www.chuatatseng.com/). (*Correspondence ) **ICML 2024, Oral Paper** **[NExT++ Research Center](https://www.nextcenter.org/), School of Computing, National University of Singapore** ----- This repository hosts the code, data and model weight of **NExT-GPT**, the first end-to-end MM-LLM that perceives input and generates output in arbitrary combinations (any-to-any) of text, image, video, and audio and beyond. **Noted**: we wrap the former old codebase into the [NExT-GPT-Lagacy](NExT-GPT-Lagacy). Please refer to this new codebase for all training and tuning procedures. ----------- ## 🎉 News - [x] [2023.09.15] 🚀🚀 Release the code of NExT-GPT in version `7b_tiva_v0`. - [x] [2023.09.27] 🔨🧩 Added modality-blended batch sampler. - [x] [2023.10.01] 📢📢 Release the T2M instruction dataset. - [x] [2023.10.04] 👏👏 Release the checkpoint of NExT-GPT in version [7b_tiva_v0](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0) . - [x] [2023.10.15] 🔨🚀 Update of NExT-GPT in version [7b_tiva_v0](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0) . - [x] [2024.10.07] 👏👏 Release the data and the corresponding construction methods, please refer [DATA_README.md](data/DATA_README.md) for more details. ## 👉 TODO - [ ] Updating NExT-GPT in more types&sizes of LLMs. - [ ] Empowering NExT-GPT with more modalities of inputs&outputs. - [ ] ... ----------- ## Example Demos Here we showcase examples generated from NExT-GPT. For more examples, kindly visit the [webpage](https://next-gpt.github.io/), or the online live [demo](https://acc414b22d6839d28f.gradio.live). https://github.com/NExT-GPT/NExT-GPT/assets/18722770/0c2b3d88-a533-4899-ab44-65580fe54538 https://github.com/NExT-GPT/NExT-GPT/assets/18722770/eb1319a6-38aa-4546-a96e-163207e7de93 https://github.com/NExT-GPT/NExT-GPT/assets/18722770/36bec0ad-9bad-4bcf-bc37-92b028f1bc6a ## Brief Introduction NExt-GPT is built on top of existing pre-trained LLM, multimodal encoder and SoTA diffusion models, with sufficient end-to-end instruction tuning.

Video-LLaMA

- **Multimodal Encoding Stage.** Leveraging established encoders to encode inputs in various modalities, where these representations are projected into language-like representations comprehensible to the LLM through a projection layer. - **LLM Understanding and Reasoning Stage.** Harnessing an existing open-sourced LLM as the core to process input information for semantic understanding and reasoning. The LLM not only directly generates text tokens but also produces unique “modality signal” tokens that serve as instructions to dictate the decoding layers whether & what modal content to output correspondingly. - **Multimodal Generation Stage.** Receiving the multimodal signals with specific instructions from LLM (if any), the Transformer-based output projection layers map the signal token representations into the ones that are understandable to following multimodal decoders. For more technical details, kindly refer to the [paper](https://arxiv.org/pdf/2309.05519.pdf). ----------- ## Getting Started ### Table of Contents: * 1. Code Structure * 2. Environment Prepar ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/next-gpt-next-gpt`](/api/graphcanon/tools/next-gpt-next-gpt) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_