Deforum Stable Diffusion
Hybrid Video in 2D/3D Animation Modes (Docs by reallybigname)
Watch the RunDiffusion overview of Deforum hosted by MattWolfe!
Overview:
- This is NOT for ‘Video Input’ animation mode.
- Hybrid video allows for video input using ‘2D’ or ‘3D’ animation_mode by mixing video into the init image (prev_img) using various compositing methods, prior to generation.
- Hybrid video motion settings allow you to use the motion from a video with or without compositing the actual video into the render.
- Hybrid color coherence options allow for frame-by-frame color coherence with video, whether mixing video in or not.
Settings Detail:
- hybrid_generate_inputframes: True | False
- Initiates extraction of video frames from your video_init_path to the inputframes folder. You only need to do this once and then you can change it to False and rerender.
- Obeys extract_nth_frame, just like normal Video Input mode.
- If overwrite_extracted_frames is True, all images in inputframes and hybridframes folders are deleted at the start of the process and regenerated.
- hybrid_use_first_frame_as_init_image: True | False
- If True, uses the first frame of the video as the init_image. False can create interesting transition effects into the video, depending on settings.
- hybrid_flow_method: DIS Medium | Farneback
- DIS Medium - DIS (Dense Inverse Search) is a dense optical flow. This setting is recommended.
- Farneback - Farneback is a sparse optical flow. Although DIS is recommended, Farneback can produce good results too. They are different.
- Enable hybrid_comp_save_extra_frames to view visualizations of the flows.
- hybrid_motion: Analyzes video frames for camera motion during generation that are used in rendering to match movement.
- None: No data will be collected or stored.
- Perspective: Uses RANSAC to make a 3x3 transformation matrix with angle, scale, rotation, and skew
- Affine: Uses RANSAC to make a 3x2 transformation matrix with angle, scale, and rotation (no skew)
- Optical Flow: Captures optical flow of entire image, uses it to warp rendering.
- *Normal 2D or 3D animation keyframing translations, zoom, rotation, depth, etc can all still be used
- But, they can be fighting against the video movement. Beware. May cause cool effects.
- Motion modes may still be used when hybrid_composite = 'False', to just transfer the motion from a video
- hybrid_motion_use_prev_img: True | False
- If enabled, changes the behavior or hybrid_motion to captures motion by comparing the current video frame to the previous rendered image, instead of the previous video frame.
- Affects RANSAC Perspective/Affine or Optical Flow.
- If your rendering varies a lot from the video, this setting could produce unpredictable results.
- hybrid_composite: True | False - Sub-options below don't do anything unless this is True.
- hybrid_comp_mask_type:
Mask types control the way that video is composited with the previous image each frame. - None: Video is mixed in without using a compositing mask.
- Depth: Composite mask uses depth maps created during rendering
- Uses the depth maps that deforum already generates.
- Requires 3D animation mod, 3D Depth Warping, and save_depth_maps = True
- Video Depth: Composite mask uses depth maps created during generation of inputframes from video
- Option must be on before generation in order to generate the depth maps.
- Blend: Composited mask is generated from a blend of video and previous image
- Use hybrid_comp_mask_blend_alpha_schedule to control the blend amount.
- Difference: Video is composited based on a difference map of video and previous image
- hybrid_comp_alpha_schedule: 0-1
- Schedule controls how much the composite video is mixed in, whether set to mask is None or using a mask.
- hybrid_comp_mask_blend_alpha_schedule: 0-1
- Schedule only used if mask type is Blend. How much of the video to blend with the previous image to create the mask. This is very different than the comp alpha, which controls the mix of whatever video compositing you select.
- hybrid_comp_mask_contrast_schedule: 0-255 - 1 is normal
- Schedule controls the contrast of the grayscale composite mask
- hybrid_comp_mask_equalize: True | False
- If True, equalizes the mask for the composite.
- hybrid_comp_mask_auto_contrast: True | False
- Auto contrast uses schedules below for it's low and high cutoff percentages.
- hybrid_comp_mask_auto_contrast_low_schedule: 0-100 - percentage
- hybrid_comp_mask_auto_contrast_high_schedule: 0-100 - percentage
- hybrid_comp_save_extra_frames: True | False
- Saves all frames generated during the hybrid video process into hybridframes folder. Depending on settings, these may include:
- video depth, composites, prev_image, masks, and optical flow visualizations
- When hybrid_comp_mask_type is Depth or Video Depth, those frames are always saved.
- hybrid_use_video_as_mse_image: True | False (currently not available in webui)
- Uses current video frame for init_mse_image every frame.
Hybrid Video Color Coherence
- color_coherence = "Video Input" matches each frame against the corresponding video frame!
- color_coherence_video_every_N_frames: Number (Default: 1)
if color_coherence is set to Video Input, this only updates color from the video every N frames - Video color coherence will work if any of these conditions are met:
- hybrid_composite is True
- hybrid_motion is Perspective, Affine, or Optical Flow
- In any of those cases, a video is available to match to.
- You can even match the color of an input video without using the video as a composite if you set composite alpha schedule to 0, or if you just use one of the motion modes.
- color_force_grayscale: True | False
- Forces images to grayscale before and after generation.
- Allows for a color_coherence mode to take effect (which can introduce color), but the image is then forced to grayscale as the init image before generation, and then forced to grayscale again after generation. This ensures the best grayscale coherence, and the color_coherence setting provides for different grayscale schemes depending on your settings (or reference video).
- The image produced is still RGB. It just has colors forced to grayscale.
- This mode is particularly effective when using the 512-depth-ema.ckpt on lower strengths. When using low strength, it is still informed by depth, but the colors can shift wildly. This is how I made this claymation look:
https://www.youtube.com/watch?v=ZPcthY0GqG8
New Seed Behaviors
- seed_behavior = "iter" (existing):
- Added seed_iter_N: Keeps same seed for N frames
- Ex. If set to 1, iterates normally like 0, 1, 2, 3, 4, 5, 6, 7, 8, 9…
- Ex. If set to 3, it will go 0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4…
- seed_behavior = "ladder":
- Increments + 2 then decrements -1 for a pattern like 0, 2, 1, 3, 2, 4, 3, 5…
- seed_behavior = "alternate":
- Increments + 1 then decrements -1 for a pattern like 0, 1, 0, 1, 0, 1, 0, 1…
Tutorials
Just one quick tutorial for now. Planning to do more soon!