Skip to main content
This node saves an encoded training dataset to disk for efficient loading during training. It takes image latents and their matching text conditioning, splits them into smaller files called shards, and stores them in a folder inside the datasets directory. It also writes a metadata file describing the dataset.

Inputs

Note: The number of items in latents must exactly match the number of items in conditioning; the node raises an error if these counts do not match. The folder_name must name a subfolder of the datasets directory (for example my_dataset) — it cannot be the datasets directory itself, and folder names that would resolve outside the datasets directory are rejected.

Outputs

This node does not produce any output data. Its function is to save files to your disk. Each shard is saved as a shard_XXXX.pkl file in the chosen folder, and a metadata.json file records the total number of samples, the number of shards, and the shard size.
This documentation was AI-generated. If you find any errors or have suggestions for improvement, please feel free to contribute! Edit on GitHub

Source fingerprint (SHA-256): 6d7b63a24ac42907b0f4a1358712cd0ed085982ecd308bce87e5376d9bbc2274