zhengchong
commited on
Commit
β’
711be4b
1
Parent(s):
9927c88
Update README.md
Browse files
README.md
CHANGED
@@ -2,8 +2,7 @@
|
|
2 |
license: cc-by-nc-sa-4.0
|
3 |
---
|
4 |
|
5 |
-
|
6 |
-
<h1 style="text-align: center;"> π CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models </h1>
|
7 |
|
8 |
<div style="display: flex; justify-content: center; align-items: center;">
|
9 |
<a href="http://arxiv.org/abs/2407.15886" style="margin: 0 2px;">
|
@@ -18,8 +17,11 @@ license: cc-by-nc-sa-4.0
|
|
18 |
<a href="http://120.76.142.206:8888" style="margin: 0 2px;">
|
19 |
<img src='https://img.shields.io/badge/Demo-Gradio-gold?style=flat&logo=Gradio&logoColor=red' alt='Demo'>
|
20 |
</a>
|
21 |
-
<a href=
|
22 |
-
<img src='https://img.shields.io/badge/
|
|
|
|
|
|
|
23 |
</a>
|
24 |
<a href="https://github.com/Zheng-Chong/CatVTON/LICENCE" style="margin: 0 2px;">
|
25 |
<img src='https://img.shields.io/badge/License-CC BY--NC--SA--4.0-lightgreen?style=flat&logo=Lisence' alt='License'>
|
@@ -27,20 +29,48 @@ license: cc-by-nc-sa-4.0
|
|
27 |
</div>
|
28 |
|
29 |
|
|
|
30 |
**CatVTON** is a simple and efficient virtual try-on diffusion model with ***1) Lightweight Network (899.06M parameters totally)***, ***2) Parameter-Efficient Training (49.57M parameters trainable)*** and ***3) Simplified Inference (< 8G VRAM for 1024X768 resolution)***.
|
31 |
|
32 |
|
|
|
33 |
## Updates
|
34 |
-
- **`2024/
|
35 |
-
- **`2024/
|
36 |
-
- **`2024/
|
37 |
-
- **`2024/
|
|
|
|
|
|
|
|
|
|
|
|
|
38 |
|
39 |
## Installation
|
40 |
-
An [Installation Guide](https://github.com/Zheng-Chong/CatVTON/INSTALL.md) is provided to help build the conda environment for CatVTON. When deploying the app, you will need Detectron2 & DensePose,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
41 |
|
42 |
-
|
43 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
44 |
|
45 |
```PowerShell
|
46 |
CUDA_VISIBLE_DEVICES=0 python app.py \
|
@@ -51,7 +81,7 @@ CUDA_VISIBLE_DEVICES=0 python app.py \
|
|
51 |
When using `bf16` precision, generating results with a resolution of `1024x768` only requires about `8G` VRAM.
|
52 |
|
53 |
## Inference
|
54 |
-
### Data Preparation
|
55 |
Before inference, you need to download the [VITON-HD](https://github.com/shadow2496/VITON-HD) or [DressCode](https://github.com/aimagelab/dress-code) dataset.
|
56 |
Once the datasets are downloaded, the folder structures should look like these:
|
57 |
```
|
@@ -66,7 +96,7 @@ Once the datasets are downloaded, the folder structures should look like these:
|
|
66 |
β β β βββ [000006_00_mask.png | 000008_00.png | ...]
|
67 |
...
|
68 |
```
|
69 |
-
For DressCode dataset, we provide [our preprocessed agnostic masks](https://drive.google.com/drive/folders/1uT88nYQl0n5qHz6zngb9WxGlX4ArAbVX?usp=share_link), download and place in `agnostic_masks` folders under each category.
|
70 |
```
|
71 |
βββ DressCode
|
72 |
| βββ test_pairs_paired.txt
|
@@ -81,8 +111,8 @@ For DressCode dataset, we provide [our preprocessed agnostic masks](https://driv
|
|
81 |
...
|
82 |
```
|
83 |
|
84 |
-
### Inference on VTIONHD/DressCode
|
85 |
-
To run the inference on the DressCode or VITON-HD dataset, run the following command, checkpoints will be
|
86 |
|
87 |
```PowerShell
|
88 |
CUDA_VISIBLE_DEVICES=0 python inference.py \
|
@@ -97,25 +127,41 @@ CUDA_VISIBLE_DEVICES=0 python inference.py \
|
|
97 |
--repaint \
|
98 |
--eval_pair
|
99 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
100 |
|
101 |
|
102 |
## Acknowledgement
|
103 |
-
Our code is modified based on [Diffusers](https://github.com/huggingface/diffusers).
|
104 |
-
|
105 |
-
|
106 |
-
and [
|
107 |
-
|
108 |
-
|
109 |
## Citation
|
110 |
|
111 |
-
```
|
112 |
@misc{chong2024catvtonconcatenationneedvirtual,
|
113 |
-
|
114 |
-
|
115 |
-
|
116 |
-
|
117 |
-
|
118 |
-
|
119 |
-
|
120 |
}
|
121 |
```
|
|
|
2 |
license: cc-by-nc-sa-4.0
|
3 |
---
|
4 |
|
5 |
+
# π CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
|
|
|
6 |
|
7 |
<div style="display: flex; justify-content: center; align-items: center;">
|
8 |
<a href="http://arxiv.org/abs/2407.15886" style="margin: 0 2px;">
|
|
|
17 |
<a href="http://120.76.142.206:8888" style="margin: 0 2px;">
|
18 |
<img src='https://img.shields.io/badge/Demo-Gradio-gold?style=flat&logo=Gradio&logoColor=red' alt='Demo'>
|
19 |
</a>
|
20 |
+
<a href="https://huggingface.co/spaces/zhengchong/CatVTON" style="margin: 0 2px;">
|
21 |
+
<img src='https://img.shields.io/badge/Space-ZeroGPU-orange?style=flat&logo=Gradio&logoColor=red' alt='Demo'>
|
22 |
+
</a>
|
23 |
+
<a href='https://zheng-chong.github.io/CatVTON/' style="margin: 0 2px;">
|
24 |
+
<img src='https://img.shields.io/badge/Webpage-Project-silver?style=flat&logo=&logoColor=orange' alt='webpage'>
|
25 |
</a>
|
26 |
<a href="https://github.com/Zheng-Chong/CatVTON/LICENCE" style="margin: 0 2px;">
|
27 |
<img src='https://img.shields.io/badge/License-CC BY--NC--SA--4.0-lightgreen?style=flat&logo=Lisence' alt='License'>
|
|
|
29 |
</div>
|
30 |
|
31 |
|
32 |
+
|
33 |
**CatVTON** is a simple and efficient virtual try-on diffusion model with ***1) Lightweight Network (899.06M parameters totally)***, ***2) Parameter-Efficient Training (49.57M parameters trainable)*** and ***3) Simplified Inference (< 8G VRAM for 1024X768 resolution)***.
|
34 |
|
35 |
|
36 |
+
|
37 |
## Updates
|
38 |
+
- **`2024/08/10`**: Our π€ [**HuggingFace Space**](https://huggingface.co/spaces/zhengchong/CatVTON) is available now! Thanks for the grant from [**ZeroGPU**](https://huggingface.co/zero-gpu-explorers)οΌ
|
39 |
+
- **`2024/08/09`**: [**Evaluation code**](https://github.com/Zheng-Chong/CatVTON?tab=readme-ov-file#3-calculate-metrics) is provided to calculate metrics π.
|
40 |
+
- **`2024/07/27`**: We provide code and workflow for deploying CatVTON on [**ComfyUI**](https://github.com/Zheng-Chong/CatVTON?tab=readme-ov-file#comfyui-workflow) π₯.
|
41 |
+
- **`2024/07/24`**: Our [**Paper on ArXiv**](http://arxiv.org/abs/2407.15886) is available π₯³!
|
42 |
+
- **`2024/07/22`**: Our [**App Code**](https://github.com/Zheng-Chong/CatVTON/blob/main/app.py) is released, deploy and enjoy CatVTON on your mechine π!
|
43 |
+
- **`2024/07/21`**: Our [**Inference Code**](https://github.com/Zheng-Chong/CatVTON/blob/main/inference.py) and [**Weights** π€](https://huggingface.co/zhengchong/CatVTON) are released.
|
44 |
+
- **`2024/07/11`**: Our [**Online Demo**](http://120.76.142.206:8888) is released π.
|
45 |
+
|
46 |
+
|
47 |
+
|
48 |
|
49 |
## Installation
|
50 |
+
An [Installation Guide](https://github.com/Zheng-Chong/CatVTON/blob/main/INSTALL.md) is provided to help build the conda environment for CatVTON. When deploying the app, you will need Detectron2 & DensePose, which are not required for inference on datasets. Install the packages according to your needs.
|
51 |
+
|
52 |
+
## Deployment
|
53 |
+
### ComfyUI Workflow
|
54 |
+
We have modified the main code to enable easy deployment of CatVTON on [ComfyUI](https://github.com/comfyanonymous/ComfyUI). Due to the incompatibility of the code structure, we have released this part in the [Releases](https://github.com/Zheng-Chong/CatVTON/releases/tag/ComfyUI), which includes the code placed under `custom_nodes` of ComfyUI and our workflow JSON files.
|
55 |
+
|
56 |
+
To deploy CatVTON to your ComfyUI, follow these steps:
|
57 |
+
1. Install all the requirements for both CatVTON and ComfyUI, refer to [Installation Guide for CatVTON](https://github.com/Zheng-Chong/CatVTON/blob/main/INSTALL.md) and [Installation Guide for ComfyUI](https://github.com/comfyanonymous/ComfyUI?tab=readme-ov-file#installing).
|
58 |
+
2. Download [`ComfyUI-CatVTON.zip`](https://github.com/Zheng-Chong/CatVTON/releases/download/ComfyUI/ComfyUI-CatVTON.zip) and unzip it in the `custom_nodes` folder under your ComfyUI project (clone from [ComfyUI](https://github.com/comfyanonymous/ComfyUI)).
|
59 |
+
3. Run the ComfyUI.
|
60 |
+
4. Download [`catvton_workflow.json`](https://github.com/Zheng-Chong/CatVTON/releases/download/ComfyUI/catvton_workflow.json) and drag it into you ComfyUI webpage and enjoy π!
|
61 |
+
|
62 |
+
> Problems under Windows OS, please refer to [issue#8](https://github.com/Zheng-Chong/CatVTON/issues/8).
|
63 |
+
>
|
64 |
+
When you run the CatVTON workflow for the first time, the weight files will be automatically downloaded, usually taking dozens of minutes.
|
65 |
|
66 |
+
|
67 |
+
<!-- <div align="center">
|
68 |
+
<img src="resource/img/comfyui.png" width="100%" height="100%"/>
|
69 |
+
</div> -->
|
70 |
+
|
71 |
+
### Gradio App
|
72 |
+
|
73 |
+
To deploy the Gradio App for CatVTON on your machine, run the following command, and checkpoints will be automatically downloaded from HuggingFace.
|
74 |
|
75 |
```PowerShell
|
76 |
CUDA_VISIBLE_DEVICES=0 python app.py \
|
|
|
81 |
When using `bf16` precision, generating results with a resolution of `1024x768` only requires about `8G` VRAM.
|
82 |
|
83 |
## Inference
|
84 |
+
### 1. Data Preparation
|
85 |
Before inference, you need to download the [VITON-HD](https://github.com/shadow2496/VITON-HD) or [DressCode](https://github.com/aimagelab/dress-code) dataset.
|
86 |
Once the datasets are downloaded, the folder structures should look like these:
|
87 |
```
|
|
|
96 |
β β β βββ [000006_00_mask.png | 000008_00.png | ...]
|
97 |
...
|
98 |
```
|
99 |
+
For the DressCode dataset, we provide [our preprocessed agnostic masks](https://drive.google.com/drive/folders/1uT88nYQl0n5qHz6zngb9WxGlX4ArAbVX?usp=share_link), download and place in `agnostic_masks` folders under each category.
|
100 |
```
|
101 |
βββ DressCode
|
102 |
| βββ test_pairs_paired.txt
|
|
|
111 |
...
|
112 |
```
|
113 |
|
114 |
+
### 2. Inference on VTIONHD/DressCode
|
115 |
+
To run the inference on the DressCode or VITON-HD dataset, run the following command, checkpoints will be automatically downloaded from HuggingFace.
|
116 |
|
117 |
```PowerShell
|
118 |
CUDA_VISIBLE_DEVICES=0 python inference.py \
|
|
|
127 |
--repaint \
|
128 |
--eval_pair
|
129 |
```
|
130 |
+
### 3. Calculate Metrics
|
131 |
+
|
132 |
+
After obtaining the inference results, calculate the metrics using the following command:
|
133 |
+
|
134 |
+
```PowerShell
|
135 |
+
CUDA_VISIBLE_DEVICES=0 python eval.py \
|
136 |
+
--gt_folder <your_path_to_gt_image_folder> \
|
137 |
+
--pred_folder <your_path_to_predicted_image_folder> \
|
138 |
+
--paired \
|
139 |
+
--batch_size=16 \
|
140 |
+
--num_workers=16
|
141 |
+
```
|
142 |
+
|
143 |
+
- `--gt_folder` and `--pred_folder` should be folders that contain **only images**.
|
144 |
+
- To evaluate the results in a paired setting, use `--paired`; for an unpaired setting, simply omit it.
|
145 |
+
- `--batch_size` and `--num_workers` should be adjusted based on your machine.
|
146 |
|
147 |
|
148 |
## Acknowledgement
|
149 |
+
Our code is modified based on [Diffusers](https://github.com/huggingface/diffusers). We adopt [Stable Diffusion v1.5 inpainting](https://huggingface.co/runwayml/stable-diffusion-inpainting) as the base model. We use [SCHP](https://github.com/GoGoDuck912/Self-Correction-Human-Parsing/tree/master) and [DensePose](https://github.com/facebookresearch/DensePose) to automatically generate masks in our [Gradio](https://github.com/gradio-app/gradio) App and [ComfyUI](https://github.com/comfyanonymous/ComfyUI) workflow. Thanks to all the contributors!
|
150 |
+
|
151 |
+
## License
|
152 |
+
All the materials, including code, checkpoints, and demo, are made available under the [Creative Commons BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. You are free to copy, redistribute, remix, transform, and build upon the project for non-commercial purposes, as long as you give appropriate credit and distribute your contributions under the same license.
|
153 |
+
|
154 |
+
|
155 |
## Citation
|
156 |
|
157 |
+
```bibtex
|
158 |
@misc{chong2024catvtonconcatenationneedvirtual,
|
159 |
+
title={CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models},
|
160 |
+
author={Zheng Chong and Xiao Dong and Haoxiang Li and Shiyue Zhang and Wenqing Zhang and Xujie Zhang and Hanqing Zhao and Xiaodan Liang},
|
161 |
+
year={2024},
|
162 |
+
eprint={2407.15886},
|
163 |
+
archivePrefix={arXiv},
|
164 |
+
primaryClass={cs.CV},
|
165 |
+
url={https://arxiv.org/abs/2407.15886},
|
166 |
}
|
167 |
```
|