Making Deforum Videos with Multiple ControlNets

  • Tadeáš Hrdina|
  • 35 steps|
  • 27 minutes

1.

Navigate to https://app.rundiffusion.com/

2.

    1. Choose "Auto1111"
    2. Click "Select"

3.

Click "Change"

4.

Large server is my choice

to handle SDXL and multiple controlnets

5.

Click "LAUNCH"

6.

Click "Settings"

7.

Click "ControlNet"

8.

  • Increase the "Multi-ControlNet: ControlNet unit number" to "5"
  • Increase the "Model cache size" to "5"

9.

Click "Apply settings"

10.

Click "Extensions"

11.

  1. disable "adetailer"
  2. disable "multidiffusion-upscale-for-automatic1111"
  3. Then click on"Apply and quit"
  4. Note you can also use the "Restart Server" option in Server Manager on the left hand menu!

your session will reboot now.

12.

Go to "Deforum" tab

13.

Inside the "Keyframes" tab

Set desired resolution and tweak the settings as you like:

Sampler,Steps,Seed and Batch name...

I set only my dimensions to 1280x768 using the sliders

For now,everything else i will leave default..

14.

  • Click the "3D" field
  • Click the "wrap" field
  • Cadence to "1" for motion intensive input videos like dancing for example,but prepare for long render time
  • Cadence to "2" or higher for input videos with lesser motion or "slowmo"which is my case.
  • Set your desired "Max frames" how long should animaton lasts?(my video input for controlnet is 24fps with 44sec duration(44x24=1056 frames)

15.

Inside the "Strenght" tab

  • set "Strenght schedule" to 0: (0)

Inside the "Motion" tab

  • Set "Translation Z" to: 0:(0)

16.

Inside the "Noise" tab

  • Select "uniform"

I like to lower default noise values. but you can leave at default

  • Set "Noise schedule" to 0: (0.025)
  • Set "Noise multiplier schedule" to 0: (1.01)

17.

Click the "Color coherence" field.

  • select "Image"

Click the "Optical flow cadence" field.

  • select "DIS Medium"

18.

**(optional)** Inside the "Coherence" tab

Click the "Color coherence" field.

  • select "Image"

that will make "Color coherence image path" field to appear below.

there you will type the path of your image which you uploaded to your storage

my color coherence image path is /mnt/private/sylv.jpg

  1. Click the "Optical flow cadence" field.
  2. Choose your desired optical flow method.

I will choose "DIS Medium"

19.

Inside the "Depth Warping & FOV" tab

  • Set "Aspect Ratio schedule" to 0: (16:9)

20.

inside the "CFG" tab

  • Set "CFG scale schedule" to 0: (5.5)

21.

Inside the "seed" tab

  • Set the Seed behavior to "fixed"

22.

Inside the "Init" tab

  • Set image init "strength" to "0"

23.

Inside the "Output" tab

  • Set your "FPS" to your input video FPS (frames per second)

in my case*:*** 24 fps

24.

set up your positive/negative prompts.

then head into "ControlNet" tab

25.

For all CNs

  • Tick "Enable"
  • Tick "Pixel Perfect"
  • Tick "My prompt is more important"
  • Set the correct path for your input video inside"ControlNet Input Video/ Image Path" field.*/mnt/private/videoinit/output.mp4*
  • Repeat the same steps for each CN Model you will enable.
  • i will be enabling 4 CN Models in total

CN Model 1:

  • Click the "Preprocessor" field.
    Select "dw_openpose_full"
  • Click the "Model" field.
    Select "thibaud_xl_openpose_256lora"
**For all CNs**

- Tick "**Enable**"

- Tick "**Pixel Perfect**"

- Tick "**My prompt is more important**"

- Set the correct path for your input video inside

  "**ControlNet Input Video/ Image Path**" field.

  `*/mnt/private/videoinit/output.mp4*`

- **Repeat** the same steps for each ***CN Model*** *you will enable.*

- i will be enabling **4 CN Models** in total

**CN Model 1:**

- Click the "**Preprocessor**" field.\
  Select "**dw_openpose_full**"
- Click the "**Model"** field.\
  Select "**thibaud_xl_openpose_256lora**"

26.

CN Model 2:

  • Tick "Enable"
  • Tick "Pixel Perfect"
  • Tick "My prompt is more important"
  • Set the correct path for your input video inside"ControlNet Input Video/ Image Path" field.*/mnt/private/videoinit/output.mp4*
  • Click the "Preprocessor" field
    Select "depth"
  • Click the "Model" field
    Select "controlnet-depth-sdxl-1.0"
**CN Model 2:**

- Tick "**Enable**"

- Tick "**Pixel Perfect**"

- Tick "**My prompt is more important**"

- Set the correct path for your input video inside

  "**ControlNet Input Video/ Image Path**" field.

  `*/mnt/private/videoinit/output.mp4*`


- Click the "**Preprocessor**" field\
  Select "**depth**"
- Click the "**Model**" field\
  Select "**controlnet-depth-sdxl-1.0**"

27.

CN Model 3

  • Tick "Enable"
  • Tick "Pixel Perfect"
  • Tick "My prompt is more important"
  • Set the correct path for your input video inside"ControlNet Input Video/ Image Path" field.*/mnt/private/videoinit/output.mp4*
  • Click the "Preprocessor" field.
    Select "lineart"
  • Click the "Model" field.
    Select "T2I-adapter_diffusers_xl_lineart"
  • Lower "Weight schedule" to 0.5
**CN Model 3**

- Tick "**Enable**"

- Tick "**Pixel Perfect**"

- Tick "**My prompt is more important**"

- Set the correct path for your input video inside

  "**ControlNet Input Video/ Image Path**" field.

  `*/mnt/private/videoinit/output.mp4*`


- Click the "**Preprocessor**" field.\
  Select "**lineart**"
- Click the "**Model**" field.\
  Select "**T2I-adapter_diffusers_xl_lineart**"
- Lower "**Weight schedule**" to **0\.5**

28.

CN Model 4

  • Tick "Enable"
  • Tick "Pixel Perfect"
  • Tick "My prompt is more important"

Set the correct path for your input video inside

  • "ControlNet Input Video/ Image Path" field.*/mnt/private/videoinit/output.mp4*
  • Click the "Preprocessor" field.
    Select "hed_safe"
  • Click the "Model" field.Select "Sargezt_xl_softedge"
  • Lower "Weight schedule" to 0.5
  • Set the "Starting Control Step schedule" to 0.35

29.

Do not forget to backup the settings!

/mnt/private/deforum_settings.txt


Click "Save Settings"

**Do not forget to backup the settings!**

*/mnt/private/deforum_settings.txt*

\
Click **"Save Settings"**

30.

Click on "Generate"

31.

Do not be alarmed by long render time! At the beginning you may see insane render times. If you are concerned, make a smaller video first!

Wait about 5-10 minutes for the estimation to stabilize..

It will drop drastically in a short time.

In my case,with my settings

around 7 hours with /cadence 1

around 3 hours with /cadence 2

**Do not be alarmed by long render time! At beggining you might see insane render times**

**wait about 5-10 minutes for the estimation to stabilize..**

It will drop drastically in a short time.

In my case,with my settings 

around **7** hours with /**cadence 1** 

around **3** hours with /**cadence 2**

32.

Click "Interrupt"

33.

  • Click the "Resume timestring" field.
    • paste the timestring which you can spot on the bottom,under the preview window
    • my timestring is 20231013180935

34.

In this case, I decided to tweak the prompt.

i can now add extra prompt on a specific keyframe, or tweak other settings/switch the checkpoint, sampler etc..

after reviewing one of 1054 frames, inside one of controlnet_inputframes folders which were automatically created by controlnet when we start generation

Carefully duplicated the prompt, then changed frame "0" to frame "232"

and changed artists and other details.

In this case,i decided to **tweak the prompt.**

i can now add extra prompt on a specific keyframe,or tweak other settings/switch the checkpoint,sampler etc..

after reviewing one of 1054 frames,inside one of **controlnet_inputframes** folders which were automaticaly created by controlnet when we start generation

carefuly duplicated the prompt,then changed frame **"0"** to frame **"232"**

and changed artistz and changed some details.

35

"Save Settings" before you make any changes !

create frequent backups so you can quickly load them any time!

About the author
Ed

Ed

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to .

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.