--- license: cc-by-nc-sa-4.0 --- # 🐈 CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
**CatVTON** is a simple and efficient virtual try-on diffusion model with ***1) Lightweight Network (899.06M parameters totally)***, ***2) Parameter-Efficient Training (49.57M parameters trainable)*** and ***3) Simplified Inference (< 8G VRAM for 1024X768 resolution)***. ## Updates - **`2024/08/10`**: Our 🤗 [**HuggingFace Space**](https://huggingface.co/spaces/zhengchong/CatVTON) is available now! Thanks for the grant from [**ZeroGPU**](https://huggingface.co/zero-gpu-explorers)! - **`2024/08/09`**: [**Evaluation code**](https://github.com/Zheng-Chong/CatVTON?tab=readme-ov-file#3-calculate-metrics) is provided to calculate metrics 📚. - **`2024/07/27`**: We provide code and workflow for deploying CatVTON on [**ComfyUI**](https://github.com/Zheng-Chong/CatVTON?tab=readme-ov-file#comfyui-workflow) 💥. - **`2024/07/24`**: Our [**Paper on ArXiv**](http://arxiv.org/abs/2407.15886) is available 🥳! - **`2024/07/22`**: Our [**App Code**](https://github.com/Zheng-Chong/CatVTON/blob/main/app.py) is released, deploy and enjoy CatVTON on your mechine 🎉! - **`2024/07/21`**: Our [**Inference Code**](https://github.com/Zheng-Chong/CatVTON/blob/main/inference.py) and [**Weights** 🤗](https://huggingface.co/zhengchong/CatVTON) are released. - **`2024/07/11`**: Our [**Online Demo**](http://120.76.142.206:8888) is released 😁. ## Installation An [Installation Guide](https://github.com/Zheng-Chong/CatVTON/blob/main/INSTALL.md) is provided to help build the conda environment for CatVTON. When deploying the app, you will need Detectron2 & DensePose, which are not required for inference on datasets. Install the packages according to your needs. ## Deployment ### ComfyUI Workflow We have modified the main code to enable easy deployment of CatVTON on [ComfyUI](https://github.com/comfyanonymous/ComfyUI). Due to the incompatibility of the code structure, we have released this part in the [Releases](https://github.com/Zheng-Chong/CatVTON/releases/tag/ComfyUI), which includes the code placed under `custom_nodes` of ComfyUI and our workflow JSON files. To deploy CatVTON to your ComfyUI, follow these steps: 1. Install all the requirements for both CatVTON and ComfyUI, refer to [Installation Guide for CatVTON](https://github.com/Zheng-Chong/CatVTON/blob/main/INSTALL.md) and [Installation Guide for ComfyUI](https://github.com/comfyanonymous/ComfyUI?tab=readme-ov-file#installing). 2. Download [`ComfyUI-CatVTON.zip`](https://github.com/Zheng-Chong/CatVTON/releases/download/ComfyUI/ComfyUI-CatVTON.zip) and unzip it in the `custom_nodes` folder under your ComfyUI project (clone from [ComfyUI](https://github.com/comfyanonymous/ComfyUI)). 3. Run the ComfyUI. 4. Download [`catvton_workflow.json`](https://github.com/Zheng-Chong/CatVTON/releases/download/ComfyUI/catvton_workflow.json) and drag it into you ComfyUI webpage and enjoy 😆! > Problems under Windows OS, please refer to [issue#8](https://github.com/Zheng-Chong/CatVTON/issues/8). > When you run the CatVTON workflow for the first time, the weight files will be automatically downloaded, usually taking dozens of minutes. ### Gradio App To deploy the Gradio App for CatVTON on your machine, run the following command, and checkpoints will be automatically downloaded from HuggingFace. ```PowerShell CUDA_VISIBLE_DEVICES=0 python app.py \ --output_dir="resource/demo/output" \ --mixed_precision="bf16" \ --allow_tf32 ``` When using `bf16` precision, generating results with a resolution of `1024x768` only requires about `8G` VRAM. ## Inference ### 1. Data Preparation Before inference, you need to download the [VITON-HD](https://github.com/shadow2496/VITON-HD) or [DressCode](https://github.com/aimagelab/dress-code) dataset. Once the datasets are downloaded, the folder structures should look like these: ``` ├── VITON-HD | ├── test_pairs_unpaired.txt │ ├── test | | ├── image │ │ │ ├── [000006_00.jpg | 000008_00.jpg | ...] │ │ ├── cloth │ │ │ ├── [000006_00.jpg | 000008_00.jpg | ...] │ │ ├── agnostic-mask │ │ │ ├── [000006_00_mask.png | 000008_00.png | ...] ... ``` For the DressCode dataset, we provide [our preprocessed agnostic masks](https://drive.google.com/drive/folders/1uT88nYQl0n5qHz6zngb9WxGlX4ArAbVX?usp=share_link), download and place in `agnostic_masks` folders under each category. ``` ├── DressCode | ├── test_pairs_paired.txt | ├── test_pairs_unpaired.txt │ ├── [dresses | lower_body | upper_body] | | ├── test_pairs_paired.txt | | ├── test_pairs_unpaired.txt │ │ ├── images │ │ │ ├── [013563_0.jpg | 013563_1.jpg | 013564_0.jpg | 013564_1.jpg | ...] │ │ ├── agnostic_masks │ │ │ ├── [013563_0.png| 013564_0.png | ...] ... ``` ### 2. Inference on VTIONHD/DressCode To run the inference on the DressCode or VITON-HD dataset, run the following command, checkpoints will be automatically downloaded from HuggingFace. ```PowerShell CUDA_VISIBLE_DEVICES=0 python inference.py \ --dataset [dresscode | vitonhd] \ --data_root_path