​
Models are essential

Models are essential building blocks for media generation workflows. They can be combined and mixed to achieve different creative effects.

The word model has many different meanings. Here, it means a data file carrying information that is required for a node graph to do its work. Specifically, it’s a data structure that models some function. As a verb, to model something means to represent it or provide an example.

The primary example of a model data file in ComfyUI is an AI diffusion model. This is a large set of data that represents the complex relationships among text strings and images, making it possible to translate words into pictures or vice versa. Other examples of common models used for image generation are multimodal vision and language models such as CLIP, and upscaling models such as RealESRGAN.

​
Model files

Model files are indispensable for generative media production. Without them, workflows cannot proceed effectively. Models are not included in the ComfyUI installation, but ComfyUI can often automatically download and install missing model files. Many models can be downloaded and installed from the ComfyUI Manager window. Models can also be found at websites such as huggingface.co, civitai.green, and github.com.

​
Using Models in ComfyUI

  1. Download and place them in the ComfyUI program directory
    1. Within the models folder, you’ll find subfolders for various types of models, such as checkpoints
    2. The ComfyUI Manager helps to automate the process of searching, downloading, and installing
    3. Restart ComfyUI if it’s running
  2. In your workflow, create the node appropriate to the model type, e.g. Load Checkpoint, Load LoRA, Load VAE
  3. In the loader node, choose the model you wish to use
  4. Connect the loader node to other nodes in your workflow

​
Adding External Model Paths

If you downloaded a model but ComfyUI cannot find it, you may need to add an additional model search path to the yaml file.

​
Example scenario

You need to add a motion LoRA and your model is located in C:\Users\YourUsername\models\animatediff_motion_lora.

​
Solution

  1. Add the below configuration to the extra_model_paths.yaml file using the correct model type property name
  2. Restart ComfyUI
my_custom_config:
  loras: C:\Users\YourUsername\models\animatediff_motion_lora/

Important: Use the correct property names for different model types:

  • loras - for LoRA models
  • checkpoints - for checkpoint models
  • vae - for VAE models
  • controlnet - for ControlNet models
  • clip - for CLIP models
  • upscale_models - for upscaling models
  • embeddings - for textual inversion embeddings
  • configs - for configuration files

For example, to add checkpoint models:

my_custom_config:
  checkpoints: C:\Users\YourUsername\models\checkpoints/

​
YAML config file location

For standard ComfyUI installations, create the extra_model_paths.yaml file directly in your ComfyUI folder (where main.py is located).

​
Desktop Application Instructions

If you’re using the ComfyUI Desktop application, the config file has a different name and location:

  • On Windows: C:\Users\YourUsername\AppData\Roaming\ComfyUI\extra_models_config.yaml
    • You can quickly navigate to this by typing %APPDATA%\ComfyUI in File Explorer address bar
  • On macOS: ~/Library/Application Support/ComfyUI/extra_models_config.yaml
    • Open Finder, press Cmd+Shift+G and enter ~/Library/Application Support/ComfyUI
  • On Linux: ~/.config/ComfyUI/extra_models_config.yaml
    • Open your file manager and navigate to the hidden folder ~/.config/ComfyUI

If the file doesn’t exist yet, you can create it yourself with any text editor.

​
File size

Models can be extremely large files relative to image files. A typical uncompressed image may require a few megabytes of disk storage. Generative AI models can be tens of thousands of times larger, up to tens of gigabytes per model. They take up a great deal of disk space and take a long time to transfer over a network.

​
Model training and refinement

A generative AI model is created by training a machine learning program on a very large set of data, such as pairs of images and text descriptions. An AI model doesn’t store the training data explicitly, but rather it stores the correlations that are implicit within the data.

Organizations and companies such as Stability AI and Black Forest Labs release “base” models that carry large amounts of generic information. These are general purpose generative AI models. Commonly, the base models need to be refined in order to get high quality generative outputs. A dedicated community of people work to refine the base models. The new, refined models produce better output, provide new or different functionality, and/or use fewer resources. Refined models can usually be run on systems with less computing power and/or memory.

​
Auxiliary models

Model functionality can be extended with auxiliary models. For example, art directing a text-to-image workflow to achieve a specific result may be difficult or impossible using a diffusion model alone. Additional models can refine a diffusion model within the workflow graph to produce desired results. Examples include LoRA (Low Rank Adaptation), a small model that is trained on a specific subject; ControlNet, a model that helps control composition using a guide image; and Inpainting, a model that allows certain diffusion models to generate new content within an existing image.