What is ControlNet ?
ControlNet is a neural network that controls image generation in Stable Diffusion by adding extra conditions. Details can be found in the article Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang and coworkers.
The first step of using ControlNet is to choose a preprocessor, it process the image so they can be a guidance for stable diffusion to understand, you can add one or more to make things more like you want to achieve by :
Depth Map
Line Art
OpenPose
or even face id to control the face
Depth
The depth preprocessor guesses the depth information from the reference image.
Depth Midas: A classic depth estimator. Also used in the Official v2 depth-to-image model.
Depth Leres: More details but also tend to render background.
Depth Leres++: Even more details.
Zoe: The level of detail sits between Midas and Leres.
Depth Anything: A newer and enhanced depth model.
example :
Line Art
Line Art renders the outline of an image. It attempts to convert it to a simple drawing.
There are a few line art preprocessors.
Line art anime: Anime-style lines
Line art anime denoise: Anime-style lines with fewer details.
Line art realistic: Realistic-style lines.
Line art coarse: Realistic-style lines with heavier weight.
example :
OpenPose
There are multiple OpenPose preprocessors.
OpenPose detects human key points such as positions of the head, shoulders, hands, etc. It is useful for copying human poses without copying other details like outfits, hairstyles, and backgrounds.
in Comfyui you can use DWOpenpose preprocessors and change as you neeed accordingly
example :
Color Palette Reference
You can color palette as reference to create a new image
example :
note : each image that already passed to Preprocessor is from the first image.
and many more..
can you help me pointing out ?
this is my first time creating an article like this, please correct me if i am wrong
for more context you can read at : https://stable-diffusion-art.com/controlnet
