base on Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On <p align="center"> <h1 align="center">Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On</h1> <h2 align="center">3DV 2024</h2> <div align="center"> <img src="./imgs/teaser.png" alt="Logo" width="100%"> </div> <div class="is-size-5 publication-authors"> <span class="author-block"> <a href="">Daiheng Gao</a>¹,</span> <span class="author-block"> <a href="">Xu Chen</a>^2,3,</span> <span class="author-block"> <a href="">Xindi Zhang</a>¹, </span> <span class="author-block"> <a href="">Qi Wang</a>¹, </span> <span class="author-block"> <a href="">Ke Sun</a>¹, </span> <span class="author-block"> <a href="">Bang Zhang</a>¹, </span> <span class="author-block"> <a href="">Liefeng Bo</a>¹, </span> <span class="author-block"> <a href="">Qixing Huang</a>⁴, </span> </div> <div class="is-size-5 publication-authors"> <span class="author-block">¹Alibaba XR Lab,</span> <span class="author-block">²ETH Zurich, Department of Computer Science,</span> <span class="author-block">³Max Planck Institute for Intelligent Systems,</span> <span class="author-block">⁴The University of Texas at Austin</span> </div> </div> <p align="center"> <br> <br></br> <a href='https://arxiv.org/abs/2308.04288'> <img src='https://img.shields.io/badge/Paper-PDF-green?style=for-the-badge&logo=arXiv&logoColor=green' alt='Paper PDF'> </a> <a href='https://tomguluson92.github.io/projects/cloth2tex/' style='padding-left: 0.5rem;'> <img src='https://img.shields.io/badge/Cloth2Tex-Page-blue?style=for-the-badge&logo=Google%20chrome&logoColor=blue' alt='Project Page'> <a href="https://www.youtube.com/watch?v=RFMNKe6supE"><img alt="youtube views" title="Subscribe to my YouTube channel" src="https://img.shields.io/youtube/views/hZd6AYin2DE?logo=youtube&labelColor=ce4630&style=for-the-badge"/></a> </p> </p> </p> </p> <br /> --- ## 1. Installation Our enviroment is *python3.8, pytorch1.13, cuda11.7*, you can change the following instructions that suitable for your settings. ```shell sudo apt-get update -y sudo apt-get install libgl1 sudo apt-get install libboost-dev ``` - **pytorch3d** [code](https://pytorch3d.org/) - **psbody-mesh** [code](https://github.com/MPI-IS/mesh) - **Kaolin** [code](https://github.com/NVIDIAGameWorks/kaolin) - **torch_geometric** [code](https://data.pyg.org/whl/) ``` pip install torch_geometric pip install pyg_lib-0.3.0+pt113cu117-cp38-cp38-linux_x86_64.whl pip install torch_cluster-1.6.1+pt113cu117-cp38-cp38-linux_x86_64.whl pip install torch_scatter-2.1.1+pt113cu117-cp38-cp38-linux_x86_64.whl pip install torch_sparse-0.6.15+pt113cu117-cp38-cp38-linux_x86_64.whl ``` --- ## 2. Architecture Cloth2Tex is composed of two phase: (1) **Coarse texture generation** and (2) **Fine texture completion**. Where Phase I is to determine the 3D garment shape and coarse texture. We do this by registering our parametric garment meshes onto catalog images using a neural mesh renderer. The pipeline’s Then Phase II is to recover fine textures from the coarse estimate of Phase I. We use image translation networks trained on large-scale data synthesized by pre-trained latent diffusion models. We only made **Phase I** publicly available for now.  --- ## 3. Inference ### Phase I (w/o automatic scaling mechanism) ``` shell python phase1_inference.py --g 1_wy --s 1.2 --d "20231017_wy" --steps_one 501 --steps_two 1001 ``` The optimized results are saved in `experiments/20231017_wy`, `x_texture_uv_1000.jpg` is the final UV texture. Users can check it with **Blender**, remember you should only reserve one material, and remove other redundant materials for textured mesh.  #### (a) reference scale coefficient The noteworthy thing here is that we are not make automatic scaling mechanism code publicly available, if you need it, you could self-implement it or manually adjust the `--s` (scale). Default coefficient for test images: ``` python per_scale_dict = {"1_wy": 1.1, "2_Polo": 0.8, # default 0.8 "3_Tshirt": 0.9, # default 0.7 "4_shorts": 0.75, # # default 0.7 "5_trousers": 0.75, "6_zipup": 1.1, "7_windcoat": 0.65, "9_jacket": 1.0, "11_skirt": 1.0} ``` #### (b) The landmark detector We are not going to release the 2D landmark detector. If you need an accurate 2D landmarks in accordance with **Cloth2Tex**, you can annotate it manually or train a simple 2D cloth landmark detector with the same definition from **Cloth2Tex**. ### Phase II (Inpainting/Completion Network) We are applying for the open-source of Phase II, we will update once approval procedure has finished. --- ## 4. Demo ### Real world 3D Try-On   Please check cloth2tex web page for animated visual results: [cloth2tex](https://tomguluson92.github.io/projects/cloth2tex/) or check our youtube video [youtube](https://www.youtube.com/watch?v=RFMNKe6supE). --- ## 5. Citation ```bibtex @article{gao2023cloth2tex, title={Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On}, author={Gao, Daiheng and Chen, Xu and Zhang, Xindi and Wang, Qi and Sun, Ke and Zhang, Bang and Bo, Liefeng and Huang, Qixing}, journal={arXiv preprint arXiv:2308.04288}, year={2023} } ```", Assign "at most 3 tags" to the expected json: {"id":"5629","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"