WanInfiniteTalkToVideo - ComfyUI Built-in Node Documentation - ComfyUI

WanInfiniteTalkToVideo generates video sequences from audio input. It uses a video diffusion model, conditioned on audio features extracted from one or two speakers, to produce a latent representation of a talking head video. The node can generate a new sequence or extend an existing one using previous frames for motion context.

Inputs

Common Inputs

Parameter	Description	Data Type	Required	Range
`mode`	The audio input mode. `single_speaker` uses one audio input. `two_speakers` enables the additional audio input and masks listed in the Two Speakers Inputs section.	DYNAMIC_COMBO	Yes	`"single_speaker"` `"two_speakers"`
`model`	The base video diffusion model.	MODEL	Yes	-
`model_patch`	The model patch containing audio projection layers.	MODEL_PATCH	Yes	-
`positive`	The positive conditioning to guide the generation.	CONDITIONING	Yes	-
`negative`	The negative conditioning to guide the generation.	CONDITIONING	Yes	-
`vae`	The VAE used for encoding images to and from the latent space.	VAE	Yes	-
`width`	The width of the output video in pixels. Must be divisible by 16. (default: 832)	INT	Yes	16 - MAX_RESOLUTION (step 16)
`height`	The height of the output video in pixels. Must be divisible by 16. (default: 480)	INT	Yes	16 - MAX_RESOLUTION (step 16)
`length`	The number of frames to generate. (default: 81)	INT	Yes	1 - MAX_RESOLUTION (step 4)
`clip_vision_output`	Optional CLIP vision output for additional conditioning.	CLIP_VISION_OUTPUT	No	-
`start_image`	Optional starting image to initialize the video sequence.	IMAGE	No	-
`audio_encoder_output_1`	The primary audio encoder output containing features for the first speaker.	AUDIO_ENCODER_OUTPUT	Yes	-
`motion_frame_count`	Number of previous frames to use as motion context. (default: 9)	INT	Yes	1 - 33
`audio_scale`	A scaling factor applied to the audio conditioning. (default: 1.0)	FLOAT	Yes	-10.0 - 10.0
`previous_frames`	Optional previous video frames to extend from. The last `motion_frame_count` frames are used as motion context.	IMAGE	No	-

Two Speakers Inputs

The inputs in this section are shown when mode is set to "two_speakers".

Parameter	Description	Data Type	Required	Range
`audio_encoder_output_2`	The second audio encoder output containing features for the second speaker.	AUDIO_ENCODER_OUTPUT	No	-
`mask_1`	Mask for the first speaker, required if using two audio inputs.	MASK	No	-
`mask_2`	Mask for the second speaker, required if using two audio inputs.	MASK	No	-

Parameter Constraints:

When mode is set to "two_speakers", audio_encoder_output_2, mask_1, and mask_2 are required for the second speaker setup.
If audio_encoder_output_2 is provided, both mask_1 and mask_2 must also be provided.
If both mask_1 and mask_2 are provided, audio_encoder_output_2 must also be provided.
If previous_frames is provided, it must contain at least as many frames as specified by motion_frame_count.

Outputs

Output Name	Description	Data Type
`model`	The patched model with audio conditioning applied.	MODEL
`positive`	The positive conditioning, potentially modified with additional context such as a start image or CLIP vision output.	CONDITIONING
`negative`	The negative conditioning, potentially modified with additional context.	CONDITIONING
`latent`	The generated video sequence in latent space.	LATENT
`trim_image`	The number of frames from the start of the motion context that should be trimmed when extending a sequence. Equals `motion_frame_count` when `previous_frames` is provided, otherwise 0.	INT

This documentation was AI-generated. If you find any errors or have suggestions for improvement, please feel free to contribute! Edit on GitHub

Source fingerprint (SHA-256): b7359490c1de86d9c82122bc227295b3b7f8a3493f629365ae0f22f9f34d9a66