Luma Image to Video 节点允许你使用Luma AI的先进技术,将静态图像转换为流畅、动态的视频内容,为图像赋予生命力和动态特性。

节点功能

此节点连接到Luma AI的图像到视频API,让用户能够基于输入图像创建动态视频。它可以理解图像中的内容并生成自然、连贯的动作,同时保持原始图像的视觉风格和特性。结合文本提示词,用户可以精确控制生成视频的动态效果。

参数说明

基本参数

参数类型默认值说明
prompt字符串""描述视频动作和内容的文本提示词
model选择项-使用的视频生成模型
resolution选择项”540p”输出视频分辨率
duration选择项-视频时长选项
loop布尔值False是否循环播放视频
seed整数0种子值,用于确定节点是否应重新运行,但实际结果与种子无关

可选参数

参数类型说明
first_image图像视频的第一帧图像(与last_image至少需要提供一个)
last_image图像视频的最后一帧图像(与first_image至少需要提供一个)
luma_conceptsLUMA_CONCEPTS用于控制相机运动和镜头效果的概念引导

参数要求

  • first_imagelast_image 至少需要提供其中一个
  • 每个图像输入(first_image和last_image)最多只接受1张图片

输出

输出类型说明
VIDEO视频生成的视频结果

使用示例

Luma Image to Video 工作流示例

Luma Image to Video 工作流教程

工作原理

Luma Image to Video 节点分析输入图像的内容和结构,然后结合文本提示词来确定如何为图像添加动态效果。它使用Luma AI的生成模型理解图像中的对象、人物或场景,并创建合理、连贯的动作序列。

用户可以通过提示词描述期望的动作类型、方向和强度,节点将据此生成相应的视频效果。通过设置不同的参数,如分辨率和时长,用户可以进一步定制输出视频的特性。

此外,通过提供起始帧和最后帧参考图像,用户可以指定视频的起始和结束状态,使动作朝特定方向发展。概念引导功能则允许用户进一步控制视频的整体风格、相机运动和美学效果。

源码参考

[节点源码 (更新于2025-05-03)]


class LumaImageToVideoGenerationNode(ComfyNodeABC):
    """
    Generates videos synchronously based on prompt, input images, and output_size.
    """

    RETURN_TYPES = (IO.VIDEO,)
    DESCRIPTION = cleandoc(__doc__ or "")  # Handle potential None value
    FUNCTION = "api_call"
    API_NODE = True
    CATEGORY = "api node/video/Luma"

    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "prompt": (
                    IO.STRING,
                    {
                        "multiline": True,
                        "default": "",
                        "tooltip": "Prompt for the video generation",
                    },
                ),
                "model": ([model.value for model in LumaVideoModel],),
                # "aspect_ratio": ([ratio.value for ratio in LumaAspectRatio], {
                #     "default": LumaAspectRatio.ratio_16_9,
                # }),
                "resolution": (
                    [resolution.value for resolution in LumaVideoOutputResolution],
                    {
                        "default": LumaVideoOutputResolution.res_540p,
                    },
                ),
                "duration": ([dur.value for dur in LumaVideoModelOutputDuration],),
                "loop": (
                    IO.BOOLEAN,
                    {
                        "default": False,
                    },
                ),
                "seed": (
                    IO.INT,
                    {
                        "default": 0,
                        "min": 0,
                        "max": 0xFFFFFFFFFFFFFFFF,
                        "control_after_generate": True,
                        "tooltip": "Seed to determine if node should re-run; actual results are nondeterministic regardless of seed.",
                    },
                ),
            },
            "optional": {
                "first_image": (
                    IO.IMAGE,
                    {"tooltip": "First frame of generated video."},
                ),
                "last_image": (IO.IMAGE, {"tooltip": "Last frame of generated video."}),
                "luma_concepts": (
                    LumaIO.LUMA_CONCEPTS,
                    {
                        "tooltip": "Optional Camera Concepts to dictate camera motion via the Luma Concepts node."
                    },
                ),
            },
            "hidden": {
                "auth_token": "AUTH_TOKEN_COMFY_ORG",
            },
        }

    def api_call(
        self,
        prompt: str,
        model: str,
        resolution: str,
        duration: str,
        loop: bool,
        seed,
        first_image: torch.Tensor = None,
        last_image: torch.Tensor = None,
        luma_concepts: LumaConceptChain = None,
        auth_token=None,
        **kwargs,
    ):
        if first_image is None and last_image is None:
            raise Exception(
                "At least one of first_image and last_image requires an input."
            )
        keyframes = self._convert_to_keyframes(first_image, last_image, auth_token)
        duration = duration if model != LumaVideoModel.ray_1_6 else None
        resolution = resolution if model != LumaVideoModel.ray_1_6 else None

        operation = SynchronousOperation(
            endpoint=ApiEndpoint(
                path="/proxy/luma/generations",
                method=HttpMethod.POST,
                request_model=LumaGenerationRequest,
                response_model=LumaGeneration,
            ),
            request=LumaGenerationRequest(
                prompt=prompt,
                model=model,
                aspect_ratio=LumaAspectRatio.ratio_16_9,  # ignored, but still needed by the API for some reason
                resolution=resolution,
                duration=duration,
                loop=loop,
                keyframes=keyframes,
                concepts=luma_concepts.create_api_model() if luma_concepts else None,
            ),
            auth_token=auth_token,
        )
        response_api: LumaGeneration = operation.execute()

        operation = PollingOperation(
            poll_endpoint=ApiEndpoint(
                path=f"/proxy/luma/generations/{response_api.id}",
                method=HttpMethod.GET,
                request_model=EmptyRequest,
                response_model=LumaGeneration,
            ),
            completed_statuses=[LumaState.completed],
            failed_statuses=[LumaState.failed],
            status_extractor=lambda x: x.state,
            auth_token=auth_token,
        )
        response_poll = operation.execute()

        vid_response = requests.get(response_poll.assets.video)
        return (VideoFromFile(BytesIO(vid_response.content)),)

    def _convert_to_keyframes(
        self,
        first_image: torch.Tensor = None,
        last_image: torch.Tensor = None,
        auth_token=None,
    ):
        if first_image is None and last_image is None:
            return None
        frame0 = None
        frame1 = None
        if first_image is not None:
            download_urls = upload_images_to_comfyapi(
                first_image, max_images=1, auth_token=auth_token
            )
            frame0 = LumaImageReference(type="image", url=download_urls[0])
        if last_image is not None:
            download_urls = upload_images_to_comfyapi(
                last_image, max_images=1, auth_token=auth_token
            )
            frame1 = LumaImageReference(type="image", url=download_urls[0])
        return LumaKeyframes(frame0=frame0, frame1=frame1)