{"data":{"slug":"haofanwang-lora-for-diffusers","name":"Lora-for-Diffusers","tagline":"Tutorial for using LoRA within Diffusers framework","github_url":"https://github.com/haofanwang/Lora-for-Diffusers","owner":"haofanwang","repo":"Lora-for-Diffusers","owner_avatar_url":"https://avatars.githubusercontent.com/u/18741068?v=4","primary_language":"Python","stars":821,"forks":50,"topics":["aigc","colossalai","diffusers","fine-tuning","guidebook","lora","stable-diffusion","stable-diffusion-webui","text-to-image"],"archived":false,"github_pushed_at":"2024-04-10T03:50:34+00:00","maintenance_label":"Dormant","stars_delta_30d":-3,"url":"https://www.graphcanon.com/tools/haofanwang-lora-for-diffusers","markdown_url":"https://www.graphcanon.com/tools/haofanwang-lora-for-diffusers.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/haofanwang-lora-for-diffusers","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=haofanwang-lora-for-diffusers","description":"The most easy-to-understand tutorial for using LoRA (Low-Rank Adaptation) within diffusers framework for AI Generation Researchers🔥","homepage_url":null,"license":"MIT","open_issues":15,"watchers":9,"ai_summary":"This repository offers a detailed tutorial focusing on implementing Low-Rank Adaptation (LoRA) techniques in the diffusers framework for facilitating AI generation research and fine-tuning tasks.","readme_excerpt":"# LoRA-for-Diffusers\n\nThis repository provides the simplest tutorial code for AIGC researchers to use Lora in just a few lines. Using this handbook, you can easily play with any Lora model from active communities such as [Huggingface](https://huggingface.co/) and [cititai](https://civitai.com/).\n\nNow, we also support [ControlNet-for-Diffusers](https://github.com/haofanwang/ControlNet-for-Diffusers), [T2I-Adapter-for-Diffusers](https://github.com/haofanwang/T2I-Adapter-for-Diffusers).\n\n# Background\n## What is Lora?\nLow-Rank Adaptation of Large Language Models ([LoRA](https://github.com/microsoft/LoRA)) is developed by Microsoft to reduce the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights. Lora attemptes to fine-tune the \"residual\" of the model instead of the entire model: i.e., train the $\\Delta W$ instead of $W$.\n\n$$\nW' = W + \\Delta W\n$$\n\nWhere $\\Delta W$ can be further decomposed into low-rank matrices : $\\Delta W = A B^T $, where $A, \\in \\mathbb{R}^{n \\times d}, B \\in \\mathbb{R}^{m \\times d}, d << n$.\nThis is the key idea of LoRA. We can then fine-tune $A$ and $B$ instead of $W$. In the end, you get an insanely small model as $A$ and $B$ are much smaller than $W$.\n\nThis training trick is quite useful for fune-tuning customized models on a large general base model. Various text to image models have been developed built on the top of the official [Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-2-1). Now, with Lora, you can efficiently train your own model with much less resources.\n\n## What is Safetensors?\nSafetensors is a new simple format for storing tensors safely (as opposed to pickle) released by Hugging Face and that is still fast (zero-copy). For its efficiency, many stable diffusion models, especially Lora models are released in safetensors format. You can find more its advantages from [huggingface/safetensors](https://github.com/huggingface/safetensors) and install it via pip install.\n\n```bash\npip install safetensors\n```\n\n# How to load Lora weights?\n\nIn this tutorial, we show to load or insert pre-trained Lora into diffusers framework. Many interesting projects can be found in [Huggingface](https://huggingface.co/) and [cititai](https://civitai.com/), but mostly in [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) framework, which is not convenient for advanced developers. We highly motivated by [cloneofsimo/lora](https://github.com/cloneofsimo/lora) about loading, merging, and interpolating trained LORAs. We mainly discuss models in safetensors format which is not well compatible with diffusers.\n\n### Full model\n\nA full model includes all modules needed (base model with or without Lora layers), they are usually stored in .ckpt or .safetensors format. We provide two examples below to show you how to use on hand.\n\n- [stabilityai/stable-diffusion-2-1](https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main) from Huggingface. \n\n- [dreamshaper](https://civitai.com/models/4384/dreamshaper) from Civitai. \n\nYou can download .ckpt or .safetensors file only. Although diffusers does not support loading them directly, they do provide the converting script. First download diffusers to local.\n\n```bash\ngit clone https://github.com/huggingface/diffusers\n```\n\n```bash\ncd ./diffusers\n\n# assume you have downloaded xxx.safetensors, it will out save_dir in diffusers format.\npython ./scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path xxx.safetensors  --dump_path save_dir --from_safetensors\n\n# assume you have downloaded xxx.ckpt, it will out save_dir in diffusers format.\npython ./scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path xxx.ckpt  --dump_path save_dir\n```\n\nThen, you can load the model\n```bash\nfrom diffusers import StableDiffusionPipeline\n\npipeline = StableDiffusionPipeline.from_pretrained(save_dir,torch_dtype=torch.float32)\n```\n\n### Lora model only\nFor n","github_created_at":"2023-02-17T14:51:56+00:00","created_at":"2026-07-11T11:40:07.979799+00:00","updated_at":"2026-09-20T06:30:32.544504+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"aigc","name":"aigc"},{"slug":"colossalai","name":"colossalai"},{"slug":"diffusers","name":"diffusers"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"lora","name":"lora"},{"slug":"stable-diffusion","name":"stable-diffusion"},{"slug":"text-to-image","name":"text-to-image"}],"trust":{"provenance":{"is_fork":false,"github_id":603067222,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-19T18:26:09.710Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":892,"last_release_at":null,"stars_delta_30d":-3,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:40:09.339Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-19T18:26:10.977Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-19T18:26:10.977Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-19T18:26:10.977Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need a straightforward tutorial to integrate LoRA techniques into diffusers for AI generation projects","Ideal if working on text-to-image tasks requiring efficient model fine-tuning without retraining entire models"],"when_not_to_use":["Not recommended if your project does not align with the diffusers framework or requires a different fine-tuning technique","Avoid if looking for comprehensive solutions beyond LoRA implementation, like end-to-end model training guides"],"source":"enrich:decision_facts","observed_at":"2026-07-14T20:45:11.696Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Detailed guide on integrating LoRA for fine-tuning with the diffusers framework in Python under MIT License"}]}}