Practical Guide to Stable Diffusion WebUI: Prompt Engineering, LoRA, VAE, and ControlNet

This article provides a hands‑on guide to using Stable Diffusion WebUI, covering prompt derivation, LoRA models, VAE models, and ControlNet, with concrete text‑to‑image and image‑to‑image examples.

Stable Diffusion is a deep‑learning text‑to‑image model; the WebUI wraps the model in an interactive interface. The most popular base model is Stable Diffusion 1.5.

Installation : Follow the official wiki at https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-NVidia-GPUs . When enabling share=True in Gradio, a frpc tunnel is created; consider disabling it for privacy.

Model types : The community (e.g., civitai.com) provides models trained via Dreambooth, LoRA, Textual Inversion, and Hypernetwork. Dreambooth yields a full new model (several GB), LoRA produces lightweight fine‑tuned adapters (tens of MB) that must be used together with the base model, and Textual Inversion and Hypernetwork behave similarly.

Prompt derivation : Upload an image, then use CLIP or DeepBooru to reverse‑engineer keywords. CLIP returns a full sentence, while DeepBooru returns a list of tags. The reverse prompt can be added to restrict unwanted elements.

LoRA usage :

Method 1 – install the sd-webui-additional-networks plugin (GitHub: https://github.com/kohya-ss/sd-web). Place LoRA files in */stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora and restart the UI.

Method 2 – copy LoRA files directly to */stable-diffusion-webui/models/Lora and restart.

If the plugin fails to install, start the UI with the insecure flag: ./webui.sh --xformers --enable-insecure-extension-access .

ControlNet : Install via the Extensions tab. Download the .pth and .yaml files from the official repository and place them in *\stable-diffusion-webui\extensions\sd-webui-controlnet\models . Restart to load.

Image‑to‑image example : Using base model revAnimated_v11 , LoRA blind_box_v1_mix , sampler Euler‑a, and an input photo, the forward prompt is: (masterpiece),(best quality), (full body:1.2), (beautiful detailed eyes), 1boy, hat, ... <lora:blindbox_v1_mix:1> The reverse prompt is (low quality:1.3), (worst quality:1.3) . Adding ControlNet with OpenPose further constrains the pose.

Text‑to‑image examples show two prompt sets with similar structure, demonstrating how LoRA trigger words and model‑specific tokens improve quality.

Prompt analysis explains the role of parentheses‑wrapped tokens (model‑specific quality boosters), LoRA trigger words, and the importance of token order for the revAnimated_v11 model.

VAE models : VAE acts as a filter or fine‑tuner. Download VAE files to /stable-diffusion-webui/models/VAE and restart. Switching VAE changes saturation and contrast, as illustrated by before/after images.

Conclusion : Stable Diffusion WebUI has a steep learning curve; understanding model characteristics, LoRA, ControlNet, and VAE is essential for reliable results. For users without high‑end hardware, commercial services like Midjourney may be more convenient.