Getting Started with ControlNet in ComfyUI: Installation, Plugins, and Workflow Guide

ComfyUI Introduction

Many users are still unfamiliar with ComfyUI, so this section gives a brief overview.

In the AI art field, Stable Diffusion is widely adopted because it is open‑source, attracting many developers and creators.

The two main tools for using Stable Diffusion are Stable Diffusion WebUI and ComfyUI.

WebUI is ready‑to‑use with many features and plugins, making it easy for beginners, but it offers limited customisation and can be harder to reproduce works via API.

ComfyUI, released later, provides strong customisation, workflow‑based automation, and better reproducibility, though it has a steeper learning curve and requires understanding of Stable Diffusion internals.

This series will cover ComfyUI concepts and usage to help you master its techniques and create unique artworks.

ComfyUI Installation and Deployment

Local Installation

Local deployment requires special network settings, an Nvidia GPU with at least 8 GB VRAM, and solid hands‑on skills; if you meet these, see the article “ComfyUI Complete Introduction: Installation and Deployment”.

Cloud Server Usage

We recommend JD Cloud’s GCS platform (https://www.jdcloud.com/cn/products/gcs). During the 618 promotion, usage costs less than ¥1 per hour, with pay‑as‑you‑go billing and optional monthly packages (≈¥680). There are also “spend‑more‑get‑more” offers for AI‑art creators.

I have published a ComfyUI image on JD Cloud that includes common workflows; it can be launched with one click, eliminating tedious setup.

Using ControlNet

Installing Plugins

If you use the cloud image mentioned above, no plugin installation is needed. For local setups, follow the steps below.

ComfyUI includes basic ControlNet nodes, but additional plugins are required for advanced capabilities.

Two recommended plugins:

ControlNet Pre‑processor Plugin – a collection of common pre‑processors. https://github.com/Fannovel16/comfyui_controlnet_aux

Advanced ControlNet – supports scheduling, masks, and other advanced features. https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet

In the ComfyUI manager, click “Install via Git URL” and paste the links to install.

After installation, restart ComfyUI and refresh the browser.

ControlNet pre‑processor models are numerous; most can be downloaded here: https://huggingface.co/lllyasviel/Annotators/tree/main

For users who have difficulty accessing Hugging Face, I have already downloaded the models to the cloud platform for direct use.

ControlNet Without Pre‑processor

ControlNet consists of three main nodes: Loader, Apply, and Reference Image.

ControlNet Loader : loads ControlNet models. ComfyUI provides two loaders – the standard “ControlNet Loader” and the “DiffControlNet Loader”, which also supports diffusers‑format models. When loading a depth ControlNet model, ensure the base Stable Diffusion model (SD‑1.5 or SDXL) matches the model version.

Reference Image : provides the visual cue (pose, depth, line‑art, etc.) for ControlNet. In the workflow example, a depth map is supplied, but a normal image can also be used with a pre‑processor.

ControlNet Apply : combines the model, reference image, and parameters to generate conditioning prompts. It exposes three adjustable parameters:

Strength – how strongly the ControlNet influences the generated image.

Start Time – the diffusion step (0‑1) when ControlNet begins to act.

End Time – the diffusion step (0‑1) when ControlNet stops influencing.

The “DiffControlNet Loader” and “Load Image” nodes feed required inputs to the Apply node via connection lines.

Inputs to ControlNet nodes :

SD Base Model – the ControlNet loader must receive a Stable Diffusion checkpoint via a “Checkpoint Loader” node.

Prompt Encoding – positive and negative text conditions, encoded by a “CLIP Text Encoder”. Multiple ControlNets can be chained to modify these conditions.

The Apply node outputs positive and negative conditions, which are then connected to a sampler to generate the final image.

When only one condition is present, the ControlNet only processes the positive side.

ControlNet With Pre‑processor

Example using a pre‑processor to extract reference information:

Three new nodes are added:

OpenPose Pose Pre‑processor : extracts human pose (body, face, hands) from the reference image, outputting a pose map. Resolution controls the shortest side of the pose image.

Perfect Pixel : calculates the optimal resolution for the pose image to match the target generation size.

Preview Image : optionally displays the pre‑processed reference image; it is not required if you already have a suitable reference.

Different pre‑processors have distinct parameter sets; refer to my earlier ControlNet series for guidance.

Parameters such as width, height, and stretch mode determine how the pose image is scaled (e.g., “Crop & Stretch” vs. “Stretch & Fill”).

Using Multiple ControlNets

Details on multi‑ControlNet workflows and configuration files are published in my AI‑art column.

Workflow Configuration Files

If the above process seems complex, you can obtain a ready‑made workflow file via enterprise WeChat for one‑click import.

Feel free to try JD Cloud’s AI compute service – no hidden fees, 50% off GPU hourly price (≈¥0.9 per hour) during the 618 promotion, with 24 GB VRAM and 83 TFLOPS FP32 performance, suitable for AIGC, large‑model inference, and rendering.

New users receive a ¥5 free GPU voucher and additional “spend‑more‑get‑more” benefits. Visit https://www.jdcloud.com/cn/products/gcs to purchase.