Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"

The framework supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B with image understanding, reasoning, and generation simultaneously. We build this repo based on LLaVA.

Release

• [05/03] 🔥 We support LLaMA3-based models! Welcome to try them here.
• [04/15] 🔥 The Hugging Face demo is available. It's a 13B-HD version, welcome to watch and try.
• [03/28] 🔥 Mini-Gemini is coming! We release the paper, demo, code, models, and data!

Contents

• Demo
• Install
• Model
• Preparation
• Train
• Evaluation
• Examples
• Citation
• Acknowledgement
• License

Demo

We provide some selected examples in this section. More examples can be found in our project page. Feel free to try our online demo!

Install

Please follow the instructions below to install the required packages.

NOTE: If you want to use the 2B version, please ensure to install the latest version Transformers (>=4.38.0).

1. Clone this repository

git clone https://github.com/dvlab-research/MGM.git

2. Install Package

conda create -n mgm python=3.10 -y
conda activate mgm
cd MGM
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install ninja
pip install flash-attn --no-build-isolation

Model

The framework is conceptually simple: dual vision encoders are utilized to provide low-resolution visual embedding and high-resolution candidates; patch info mining is proposed to conduct patch-level mining between high-resolution regions and low-resolution visual queries; LLM is utilized to marry text with images for both comprehension and generation at the same time.

We provide all our fully finetuned models on Stage 1 and 2 data:

Model	LR	HR	Base LLM	Vision Encoder	Finetuning Data	Finetuning schedule	Download
MGM-2B	336	768	Gemma-2B	CLIP-L	MGM-Instruct	full_ft-1e	ckpt
MGM-7B	336	768	Vicuna-7B-v1.5	CLIP-L	MGM-Instruct	full_ft-1e	ckpt
MGM-13B	336	768	Vicuna-13B-v1.5	CLIP-L	MGM-Instruct	full_ft-1e	ckpt
MGM-8B	336	768	LLaMA-3-8B-Instruct