Flux ControlNet - Ultimate Workflow

Video guide https://youtu.be/B3HBAiGuySk

In this guide we’ll go over a workflow for using Flux ControlNet within Comfy. You will be able to easily use one or more ControlNets just by flipping a switch. ControlNet will control your output image by the input image you load.

If you don't have Comfy or the Comfy Manager installed, you can find a guide on it here: https://www.youtube.com/watch?v=KTPLOqAMR0s

Drag and drop the .json file (download at bottom) into your Comfy. If you get red nodes, make sure to go into your Manager and Install all missing custom nodes. 

Install Flux ControlNet models by clicking Model Manager and then search for ControlNet Union Pro. Use the larger one if you don't have vram or storage issues. The smaller fp8 one can be used for lower end machines. These models go in the /models/ControlNet/ folder

First, load your preferred image in the Load image node and select the size you want your output image to be. Your input image will be resized accordingly.

Select your preferred model GGUF model. By default in this workflow we're using Flux Dev Q8 GGUF, which is the quant with highest quality. If you don't have GGUF models, you can find them in the Model Manager (as seen above for ControlNets, search for GGUF). These models go in the /models/unet/ folder

The VAE and CLIP can also be found through the model manager.
Also available at this link: https://huggingface.co/comfyanonymous/flux_text_encoders/tree/main
ae.safetensors go in /models/vae/
tx5xxl_fp16 and clip_l go in /models/clip/

If you want to load a Lora, you can select these from the Load Lora nodes. To disable or enable loras, right click a node and select “Bypass node” or select it and press Ctrl + B / Cmd + B. When they are purple, they are disabled.

Type your prompt in the positive (green) node. This is where you type what you want to see in your image. For this Flux workflow, negative prompting is not enabled.

These nodes are your ControlNets and they are split up into several groups. First, a ControlNet model is loaded. We’re also loading a preprocessor to look at the input image and send that image preprocessed together with the ControlNet model into the Apply ControlNet node. This is where you would finetune your settings if needed.

Recommended values for Flux dev ControlNet Union Pro:
Strength around 0.5-0.7
End_percent around 0.1-0.2

Strength: How much impact this ControlNet will have on your result, set 0-1

Start_percent: When in the generation ControlNet will activate, set 0-1

End_percent: When in the generation ControlNet will finish, set 0-1

Example A: If your generation is running for 20 steps (Ksampler value) and you set end_percent to 0.5, then ControlNet will be active for 50% of that generation time, 10 steps. The last 10 steps will run without ControlNet.

Example B: Your generation is set at 20 steps in the Ksampler. Start_percent is set at 0.5 and end_percent at 0.8. Your generation will run 10 steps without ControlNet. It will activate after 10 steps and run with ControlNet and then disable again after 16 steps to finish the last 4 steps without ControlNet.

To enable or disable a ControlNet group, click the “Fast Bypasser” node in the right corner which says Enable yes/no

Your output will be in this group together with the Ksampler. The Ksampler is where the actual generation or rendering takes place. 

Set the amount of steps you want.
Around 10-20 for euler

CFG is based on the model you use. We're using Flux so set CFG at 1

SD 1.5 models use between 3-7, but specialty models like Flux, SD-LCM/Lightning/Turbo etc. will use a lower CFG like 1-2. Check your model information.

Denoise is how much of your input image will change. This is not based on ControlNet, but more similar to img2img. If you lower the denoise value, you will retain parts of your latent_image input. However in this workflow our latent_image input is just an empty node so nothing will be retained. Keep value at 1.

Seed is set at randomize, and gives you a random image each time you click generate. If you want to finetune your settings, you can set this to fixed to see the changes you make for each generation.

As an added bonus, there is an ultimate SD Upscale node to upscale your results. Recommended is to leave this disabled at first, as the extra generation time will be quite resource intensive. 

Feel free to play with the values to adjust the upscaling, try different upscaling models, steps and samplers to find the settings for your image. 

It currently is upscaling in tiles. Recommended value for tiles in Flux are 1024x1024
You can download the NOMOS Upscaler model at https://openmodeldb.info/models/4x-Nomos8kDAT
Place in your /models/ESRGAN/ folder

TROUBLESHOOTING

Model selection nodes turn red
Make sure your models are in the right folder. Refresh ComfyUI and re-select the model in the dropdown list of the red border node.

Image is weird or ugly
Turn down ControlNet strength or end_percent

Workflow crashes or goes out of memory
Load fp8 ControlNet Union Pro.
Load fp8 text encoders.
Load a smaller Flux dev GGUF model (like Q1-5)
Load fewer ControlNets (disable on the switch)