LoRA Model Training Guide for Stable Diffusion: Comparison, Workflow, and Tips

With the rise of large‑model AI, AIGC has become accessible to both algorithm engineers and artists. The open‑source nature of Stable Diffusion and the availability of models on CivitAI have made AI image generation popular, and training a custom model is now feasible for most users.

The article first compares different fine‑tuning methods for Stable Diffusion, including Dreambooth, Hypernetwork, Textual Inversion, and LoRA. A table summarizes their training size, minimum GPU memory, and typical training time, showing that LoRA offers the best trade‑off in both model size and training duration.

A second table lists the advantages, disadvantages, and suitable scenarios for each method, highlighting LoRA’s flexibility, low cost, and strong compatibility, while noting its susceptibility to style interference from the base model.

The core of the guide is a step‑by‑step workflow for training a LoRA model (using an example character “Onion Head”):

Dataset preparation : collect at least 20 high‑quality images of the target character from multiple angles. If images are blurry, use the Stable Diffusion “extras” tab to upscale and restore quality. Recommended upscalers: ESRGAN_4x for faces/realistic images, R‑ESRGAN 4x+ Anime6B for anime‑style images.

Dataset preprocessing : flip images to augment data, split overly large pictures, apply automatic focus cropping (especially for faces), generate textual tags with BLIP (realistic images) or deepbooru (anime images), and optionally add custom tags (e.g., prepend yangcongtou to each prompt).

Parameter adjustment : set LoRA weight (typically 0.7‑0.9) and other training hyper‑parameters.

Training : with a 11 GB GPU, training 20 epochs on 20 images usually finishes within 20 minutes.

Prediction : select the base model, load the trained LoRA, and adjust the LoRA weight to generate images featuring the custom tag.

The guide also provides practical advice: ensure the dataset is clear, diverse, and properly sized; choose a base model that matches the desired style; tune training parameters (epochs, learning rate, schedule) carefully; keep training and inference image dimensions consistent; and iterate by augmenting the dataset with generated images.

Additional resources include a GitHub repository for LoRA scripts ( https://github.com/Akegarasu/lora-scripts ) and reference videos/articles.