{"data":{"slug":"jia-lab-research-mgm","name":"MGM","tagline":"Official repo for 'Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models'","github_url":"https://github.com/JIA-Lab-research/MGM","owner":"JIA-Lab-research","repo":"MGM","owner_avatar_url":"https://avatars.githubusercontent.com/u/64006090?v=4","primary_language":"Python","stars":3330,"forks":275,"topics":["generation","large-language-models","vision-language-model"],"archived":false,"github_pushed_at":"2024-05-04T14:36:51+00:00","url":"https://www.graphcanon.com/tools/jia-lab-research-mgm","markdown_url":"https://www.graphcanon.com/tools/jia-lab-research-mgm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jia-lab-research-mgm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jia-lab-research-mgm","description":"Official repo for \"Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models\"","homepage_url":null,"license":"Apache-2.0","open_issues":61,"watchers":26,"ai_summary":"A Python-based framework supporting dense and MoE Large Language Models (LLMs) from 2B to 34B, focusing on image understanding, reasoning, and generation. Built based on LLaVA.","readme_excerpt":"# Official repo for \"Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models\"\n\n\n\n\n\n\n\n\n\nThe framework supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B with image understanding, reasoning, and generation simultaneously. We build this repo based on LLaVA.\n\n## Release\n- [05/03] 🔥 We support LLaMA3-based models! Welcome to try them [here](https://huggingface.co/collections/YanweiLi/mgm-6603c50b9b43d044171d0854).\n- [04/15] 🔥 The [Hugging Face demo](https://huggingface.co/spaces/wcy1122/MGM) is available. It's a 13B-HD version, welcome to watch and try.\n- [03/28] 🔥 Mini-Gemini is coming! We release the [paper](https://arxiv.org/pdf/2403.18814.pdf), [demo](http://103.170.5.190:7860/), [code](https://github.com/dvlab-research/MGM), [models](https://huggingface.co/collections/YanweiLi/mgm-6603c50b9b43d044171d0854'), and [data](https://huggingface.co/collections/YanweiLi/mgm-data-660463ea895a01d8f367624e)!\n\n## Contents\n- [Demo](#demo)\n- [Install](#install)\n- [Model](#model)\n- [Preparation](#preparation)\n- [Train](#train)\n- [Evaluation](#evaluation)\n- [Examples](#examples)\n- [Citation](#citation)\n- [Acknowledgement](#acknowledgement)\n- [License](#license)\n\n## Demo\nWe provide some selected examples in this section. More examples can be found in our [project page](https://mini-gemini.github.io/). Feel free to try our online [demo](http://103.170.5.190:7860/)!\n\n
\n\n
\n\n## Install\nPlease follow the instructions below to install the required packages.\n\nNOTE: If you want to use the 2B version, please ensure to install the latest version Transformers (>=4.38.0).\n\n1. Clone this repository\n```bash\ngit clone https://github.com/dvlab-research/MGM.git\n```\n\n2. Install Package\n```bash\nconda create -n mgm python=3.10 -y\nconda activate mgm\ncd MGM\npip install --upgrade pip # enable PEP 660 support\npip install -e .\n```\n\n3. Install additional packages for training cases\n```bash\npip install ninja\npip install flash-attn --no-build-isolation\n```\n\n## Model\nThe framework is conceptually simple: dual vision encoders are utilized to provide low-resolution visual embedding and high-resolution candidates;\npatch info mining is proposed to conduct patch-level mining between high-resolution regions and low-resolution visual queries;\nLLM is utilized to marry text with images for both comprehension and generation at the same time.\n\n
\n\n
\n\nWe provide all our fully finetuned models on Stage 1 and 2 data:\n\n| Model | LR | HR | Base LLM | Vision Encoder | Finetuning Data | Finetuning schedule | Download |\n|----------|----------|----------|----------|----------------|---------------|--------------------|------------------|\n| MGM-2B | 336 | 768 | Gemma-2B | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/MGM-2B) |\n| MGM-7B | 336 | 768 | Vicuna-7B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/MGM-7B) |\n| MGM-13B | 336 | 768 | Vicuna-13B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/MGM-13B) |\n| MGM-8B | 336 | 768 | LLaMA-3-8B-Instruct ","github_created_at":"2024-03-26T14:48:45+00:00","created_at":"2026-07-07T17:35:51.534667+00:00","updated_at":"2026-07-07T20:02:31.203739+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"generation","name":"generation"},{"slug":"vision-language-model","name":"vision-language-model"},{"slug":"large-language-models","name":"large-language-models"}]}}