---
title: "MGM"
type: "tool"
slug: "jia-lab-research-mgm"
canonical_url: "https://www.graphcanon.com/tools/jia-lab-research-mgm"
github_url: "https://github.com/JIA-Lab-research/MGM"
homepage_url: null
stars: 3330
forks: 275
primary_language: "Python"
license: "Apache-2.0"
categories: ["llm-frameworks", "model-training"]
tags: ["generation", "vision-language-model", "large-language-models"]
updated_at: "2026-07-07T20:02:31.203739+00:00"
---
# MGM
> Official repo for 'Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models'
A Python-based framework supporting dense and MoE Large Language Models (LLMs) from 2B to 34B, focusing on image understanding, reasoning, and generation. Built based on LLaVA.
## Facts
- Repository: https://github.com/JIA-Lab-research/MGM
- Stars: 3,330 · Forks: 275 · Open issues: 61 · Watchers: 26
- Primary language: Python
- License: Apache-2.0
- Last pushed: 2024-05-04T14:36:51+00:00
## Categories
- [LLM Frameworks](/categories/llm-frameworks.md)
- [Model Training](/categories/model-training.md)
## Tags
generation, vision-language-model, large-language-models
## Related tools
- [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,664)
- [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,025)
- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,350)
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,582)
- [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702)
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,715)
- [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,619)
- [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,189)
## README (excerpt)
```text
# Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"
The framework supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B with image understanding, reasoning, and generation simultaneously. We build this repo based on LLaVA.
## Release
- [05/03] 🔥 We support LLaMA3-based models! Welcome to try them [here](https://huggingface.co/collections/YanweiLi/mgm-6603c50b9b43d044171d0854).
- [04/15] 🔥 The [Hugging Face demo](https://huggingface.co/spaces/wcy1122/MGM) is available. It's a 13B-HD version, welcome to watch and try.
- [03/28] 🔥 Mini-Gemini is coming! We release the [paper](https://arxiv.org/pdf/2403.18814.pdf), [demo](http://103.170.5.190:7860/), [code](https://github.com/dvlab-research/MGM), [models](https://huggingface.co/collections/YanweiLi/mgm-6603c50b9b43d044171d0854'), and [data](https://huggingface.co/collections/YanweiLi/mgm-data-660463ea895a01d8f367624e)!
## Contents
- [Demo](#demo)
- [Install](#install)
- [Model](#model)
- [Preparation](#preparation)
- [Train](#train)
- [Evaluation](#evaluation)
- [Examples](#examples)
- [Citation](#citation)
- [Acknowledgement](#acknowledgement)
- [License](#license)
## Demo
We provide some selected examples in this section. More examples can be found in our [project page](https://mini-gemini.github.io/). Feel free to try our online [demo](http://103.170.5.190:7860/)!
## Install
Please follow the instructions below to install the required packages.
NOTE: If you want to use the 2B version, please ensure to install the latest version Transformers (>=4.38.0).
1. Clone this repository
```bash
git clone https://github.com/dvlab-research/MGM.git
```
2. Install Package
```bash
conda create -n mgm python=3.10 -y
conda activate mgm
cd MGM
pip install --upgrade pip # enable PEP 660 support
pip install -e .
```
3. Install additional packages for training cases
```bash
pip install ninja
pip install flash-attn --no-build-isolation
```
## Model
The framework is conceptually simple: dual vision encoders are utilized to provide low-resolution visual embedding and high-resolution candidates;
patch info mining is proposed to conduct patch-level mining between high-resolution regions and low-resolution visual queries;
LLM is utilized to marry text with images for both comprehension and generation at the same time.
We provide all our fully finetuned models on Stage 1 and 2 data:
| Model | LR | HR | Base LLM | Vision Encoder | Finetuning Data | Finetuning schedule | Download |
|----------|----------|----------|----------|----------------|---------------|--------------------|------------------|
| MGM-2B | 336 | 768 | Gemma-2B | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/MGM-2B) |
| MGM-7B | 336 | 768 | Vicuna-7B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/MGM-7B) |
| MGM-13B | 336 | 768 | Vicuna-13B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/MGM-13B) |
| MGM-8B | 336 | 768 | LLaMA-3-8B-Instruct
```
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/jia-lab-research-mgm`](/api/graphcanon/tools/jia-lab-research-mgm)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_
