Self-Adaptive Clothing Mapping Based Virtual Try-on
Article No.: 61, Pages 1 - 26
Abstract
VTON (Virtual Try-ON), as an innovative visual application in e-commerce scenarios with great commercial value, has been widely studied in recent years. Due to its better robustness and realistic effect, deformation-synthesize-based VTON has become the dominant approach in this field. Existing clothing deformation techniques optimize the mapping relations between the original clothing image and the ground truth (GT) image of the worn clothing. However, there are color differences between the original and GT clothing images caused by lighting, warping, and occlusion. The color differences may lead to misaligned clothing mapping by only minimizing the cost of pixel value difference. Another drawback is that taking the parsing prediction as GT will bring alignment remnant, rooting in the processing order of parsing and deformation. Aiming above two drawbacks, we put forward SAME-VTON (Self-Adaptive clothing Mapping basEd Virtual Try-ON) for achieving realistic virtual try-on results. The core of SAME-VTON is the self-adaptive clothing mapping technique, composed of two parts: a color-adaptive clothing mapping module and a parsing-adaptive prediction process. In the color-adaptive clothing mapping module, we map each pixel of the target clothing with a combination of multiple pixel values from the original clothing image, which considers both the position and color changes. Furthermore, different combination weights are learned to increase the diversity of color mapping. In the parsing-adaptive prediction process, the color-adaptive clothing mapping module is adopted to deform clothing first, then the human parsing result is predicted under the reference of the deformed clothing, which can avoid alignment remnant. Extensive experiments demonstrate that the proposed SAME-VTON with the self-adaptive clothing mapping technique can achieve optimal mapping in the case of large color differences and obtain superior results compared with existing deformation-synthesize-based VTON.
References
[1]
Wen-Huang Cheng, Sijie Song, Chieh-Yun Chen, Shintami Chusnul Hidayati, and Jiaying Liu. 2021. Fashion meets computer vision: A survey. ACM Comput. Surv. 54, 4 (July 2021), Article 72, 41 pages. DOI:
[2]
Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. 2021. VITON-HD: High-resolution virtual try-on via misalignment-aware normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14131–14140.
[3]
Ayush Chopra, Rishabh Jain, Mayur Hemani, and Balaji Krishnamurthy. 2021. ZFlow: Gated appearance flow-based virtual try-on with 3D priors. In Proceedings of the IEEE International Conference on Computer Vision. 5433–5442.
[4]
Aiyu Cui, Daniel McKee, and Svetlana Lazebnik. 2021. Dressing in order: Recurrent person image generation for pose transfer, virtual try-on and outfit editing. In Proceedings of the IEEE International Conference on Computer Vision. 14638–14647.
[5]
Lavinia De Divitiis, Federico Becattini, Claudio Baecchi, and Alberto Del Bimbo. 2023. Disentangling features for fashion recommendation. ACM Trans. Multimedia Comput. Commun. Appl. 19, 1s (Jan. 2023), Article 39, 21 pages. DOI:
[6]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248–255. DOI:
[7]
Foivos I. Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. 2020. ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS J. Photogramm. Remote Sens. 162 (April2020), 94–114. DOI:
[8]
Benjamin Fele, Ajda Lampe, Peter Peer, and Vitomir Struc. 2022. C-VTON: Context-driven image-based virtual try-on network. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3144–3153.
[9]
Zunlei Feng, Zhenyun Yu, Yongcheng Jing, Sai Wu, Mingli Song, Yezhou Yang, and Junxiao Jiang. 2019. Interpretable partitioned embedding for intelligent multi-item fashion outfit composition. ACM Trans. Multimedia Comput. Commun. Appl. 15, 2s (July 2019), Article 61, 20 pages. DOI:
[10]
Matteo Fincato, Marcella Cornia, Federico Landi, Fabio Cesari, and Rita Cucchiara. 2022. Transform, warp, and dress: A new transformation-guided model for virtual try-on. ACM Trans. Multimedia Comput. Commun. Appl. 18, 2 (Feb. 2022), Article 62, 24 pages. DOI:
[11]
Matteo Fincato, Federico Landi, Marcella Cornia, Fabio Cesari, and Rita Cucchiara. 2021. VITON-GT: An image-based virtual try-on model with geometric transformations. In Proceedings of the International Conference on Pattern Recognition. 7669–7676. DOI:
[12]
Xin Gao, Zhenjiang Liu, Zunlei Feng, Chengji Shen, Kairi Ou, Haihong Tang, and Mingli Song. 2021. Shape controllable virtual try-on for underwear models. In Proceedings of the 29th ACM International Conference on Multimedia (MM’21). ACM, New York, NY, 563–572. DOI:
[13]
Chongjian Ge, Yibing Song, Yuying Ge, Han Yang, Wei Liu, and Ping Luo. 2021. Disentangled cycle consistency for highly-realistic virtual try-on. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16928–16937.
[14]
Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. 2021. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8485–8493.
[15]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems 27 (NIPS ’14).
[16]
Xiaoling Gu, Jie Huang, Yongkang Wong, Jun Yu, Jianping Fan, Pai Peng, and Mohan S. Kankanhalli. 2022. PAINT: Photo-realistic fashion design synthesis. ACM Trans. Multimedia Comput. Commun. Appl. Accepted June 2022. DOI:
[17]
Xintong Han, Weilin Huang, Xiaojun Hu, and Matthew Scott. 2019. ClothFlow: A flow-based model for clothed person generation. In Proceedings of the IEEE International Conference on Computer Vision. 10470–10479. DOI:
[18]
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S. Davis. 2018. VITON: An image-based virtual try-on network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. DOI:arxiv:1711.08447
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778. DOI:
[20]
Zhu Heming, Cao Yu, Jin Hang, Chen Weikai, Du Dong, Wang Zhangye, Cui Shuguang, and Han Xiaoguang. 2020. Deep Fashion3D: A dataset and benchmark for 3D garment reconstruction from single images. In Proceedings of the European Conference on Computer Vision. 512–530.
[21]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). 6629–6640.
[22]
Alain Horé and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In Proceedings of the International Conference on Pattern Recognition. 2366–2369. DOI:
[23]
Chia-Wei Hsieh, Chieh-Yun Chen, Chien-Lung Chou, Hong-Han Shuai, Jiaying Liu, and Wen-Huang Cheng. 2019. FashionOn: Semantic-guided image-based virtual try-on with detailed human and clothing information. In Proceedings of the 27th ACM International Conference on Multimedia (MM’19). ACM, New York, NY, 275–283. DOI:
[24]
Phillip Isola, Jun-Yan Zhu, Tinghui “Zhou”, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[25]
Thibaut Issenhuth, Jérémie Mary, and Clément Calauzènes. 2020. Do not mask what you do not need to mask: A parser-free virtual try-on. In Computer Vision—ECCV 2020. Lecture Notes in Computer Science, Vol. 12365. Springer, 619–635. DOI:
[26]
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with Gumbel-softmax. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkE3y85ee
[27]
Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. 2020. BCNet: Learning body and cloth shape from a single image. In Proceedings of the European Conference on Computer Vision.
[28]
Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. 2021. Amalgamating knowledge from heterogeneous graph neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15709–15718.
[29]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations. http://arxiv.org/abs/1412.6980
[30]
Kathleen M. Lewis, Srivatsan Varadharajan, and Ira Kemelmacher-Shlizerman. 2021. TryOnGAN: Body-aware try-on via layered interpolation. ACM Trans. Graph. 40, 4 (2021), Article 115, 10 pages.
[31]
Kathleen M. Lewis, Srivatsan Varadharajan, and Ira Kemelmacher-Shlizerman. 2021. VOGUE: Try-on by StyleGAN interpolation optimization. arXiv preprint arXiv:2101.02285 (2021).
[32]
Kedan Li, Min Jin Chong, Jeffrey Zhang, and Jingen Liu. 2021. Toward accurate and realistic outfits visualization with attention to details. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15546–15555.
[33]
Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. 2022. Self-correction for human parsing. IEEE Trans. Pattern Anal. Mach. Intell. 44, 6 (2022), 3260–3271. DOI:
[34]
Luoqi Liu, Junliang Xing, Si Liu, Hui Xu, Xi Zhou, and Shuicheng Yan. 2014. “Wow! you are so beautiful today!” ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s (Oct. 2014), Article 20, 22 pages. DOI:
[35]
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. 2023. Slimmable dataset condensation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3759–3768.
[36]
Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. 2020. Learning to dress 3D people in generative clothing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[37]
Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian. 2020. Controllable person image synthesis with attribute-decomposed GAN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[38]
Matiur Rahman Minar, Thai Thanh Tuan, Heejune Ahn, Paul Rosin, and Yu-Kun Lai. 2020. 3D reconstruction of clothes using a human body model and its application to image-based virtual try-on. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop.
[39]
Matiur Rahman Minar, Thai Thanh Tuan, Heejune Ahn, Paul L. Rosin, and Yu-Kun Lai. 2020. CP-VTON+: Clothing shape and texture preserving image-based virtual try-on. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop. 4.
[40]
Aymen Mir, Thiemo Alldieck, and Gerard Pons-Moll. 2020. Learning to transfer texture from clothing images to 3D humans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[41]
Davide Morelli, Matteo Fincato, Marcella Cornia, Federico Landi, Fabio Cesari, and Rita Cucchiara. 2022. Dress code: High-resolution multi-category virtual try-on. In Proceedings of the European Conference on Computer Vision.
[42]
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. 2022. On aliased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[43]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NIPS’19). 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[44]
Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. TailorNet: Predicting clothing in 3D as a function of human pose, shape and garment style. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[45]
Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H. Li, and Ge Li. 2020. Deep image spatial transformation for person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7687–7696. DOI:
[46]
Iasonas Kokkinos R{i}za Alp Güler, Natalia Neverova. 2018. DensePose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[47]
Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. 2017. Convolutional neural network architecture for geometric matching In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.DOI:
[48]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Springer, 234–241. DOI:
[49]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training GANs. arXiv:cs.LG/1606.03498 (2016).
[50]
Igor Santesteban, Nils Thuerey, Miguel A. Otaduy, and Dan Casas. 2021. Self-supervised collision handling via generative 3D garment models for virtual try-on. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[51]
Scott Schaefer, Travis McPhail, and Joe Warren. 2006. Image deformation using moving least squares. ACM Trans. Graph. 25, 3 (2006), 533–540.
[52]
Yu Shen, Junbang Liang, and Ming C. Lin. 2020. GAN-based garment generation using sewing pattern images. In Proceedings of the European Conference on Computer Vision.
[53]
K. Simonyan and A. Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations.
[54]
Jie Song, Ying Chen, Jingwen Ye, and Mingli Song. 2022. Spot-adaptive knowledge distillation. IEEE Trans. Image Process. 31 (2022), 3359–3370.
[55]
Olga Sorkine and Marc Alexa. 2007. As-rigid-as-possible surface modeling. In Proceedings of the Symposium on Geometry Processing, Vol. 4. 109–116.
[56]
Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. 2020. SIZER: A dataset and model for parsing 3D clothing and learning size sensitive 3D clothing. In Proceedings of the European Conference on Computer Vision.
[57]
Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. 2018. Toward characteristic-preserving image-based virtual try-on network. In Proceedings of the European Conference on Computer Vision. 589–604.
[58]
Jiahang Wang, Wei Zhang, Weizhong Liu, and Tao Mei. 2020. Down to the last detail: Virtual try-on with detail carving. arXiv:1912.06324 [Cs] (2020).
[59]
Z. Wang. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612.
[60]
Simon N. Wood. 2003. Thin plate regression splines. J. R. Stat. Soc. Soc. B 65, 1 (2003), 95–114.
[61]
Zhenyu Xie, Zaiyu Huang, Fuwei Zhao, Haoye Dong, Michael Kampffmeyer, and Xiaodan Liang. 2021. Towards scalable unpaired virtual try-on via patch-routed spatially-adaptive GAN. In Advances in Neural Information Processing Systems (NIPS’21).
[62]
Zhenyu Xie, Xujie Zhang, Fuwei Zhao, Haoye Dong, Michael C. Kampffmeyer, Haonan Yan, and Xiaodan Liang. 2021. WAS-VTON: Warping architecture search for virtual try-on network. In Proceedings of the ACM International Conference on Multimedia (MM’21). 3350–3359.
[63]
Cheng Xu, Zejun Chen, Jiajie Mai, Xuemiao Xu, and Shengfeng He. 2022. Pose and attribute consistent person image synthesis. ACM Trans. Multimedia Comput. Commun. Appl. 19, 2s (July 2022), Article 81, 21 pages. DOI:
[64]
Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. 2020. Towards photo-realistic virtual try-on by adaptively generating\(\leftrightarrow\)preserving image content. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[65]
Xin Yang, Xuemeng Song, Fuli Feng, Haokun Wen, Ling-Yu Duan, and Liqiang Nie. 2021. Attribute-wise explainable fashion compatibility modeling. ACM Trans. Multimedia Comput. Commun. Appl. 17, 1 (April 2021), Article 36, 21 pages. DOI:
[66]
Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. 2022. Deep model reassembly. In Advances in Neural Information Processing Systems 35 (NIPS’22).25739–25753.
[67]
Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. 2019. VTNFP: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE International Conference on Computer Vision. 10510–10519. DOI:
[68]
Fuwei Zhao, Zhenyu Xie, Michael Kampffmeyer, Haoye Dong, Songfang Han, Tianxiang Zheng, Tao Zhang, and Xiaodan Liang. 2021. M3D-VTON: A monocular-to-3D virtual try-on network. In Proceedings of the IEEE International Conference on Computer Vision. 13239–13249.
Index Terms
- Self-Adaptive Clothing Mapping Based Virtual Try-on
Recommendations
Customized 3D Clothes Modeling for Virtual Try-on System based on Multiple Kinects
GRAPP 2016: Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications: Volume 1: GRAPPMost existing 3D virtual try-on systems put clothes designed in one environment on a human captured in another environment, which cause the mismatching brightness problem. And also typical 3D clothes modeling starts with manually designed 2D patterns, ...
Interactive virtual try-on clothing design systems
Clothing computer design systems include three integrated parts: garment pattern design in 2D/3D, virtual try-on and realistic clothing simulation. Some important results have been obtained in pattern design and clothing simulation since the 1980s. ...
The development of a clothing fit evaluation system under virtual environment
This study proposes an innovative clothing fit evaluation method for virtual try- on using 3D digital human model. The clothing patterns are first designed and subsequently sewn together around a human model. Because complex cloth shapes are difficult ...
Comments
Information & Contributors
Information
Published In
March 2024
665 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 23 October 2023
Online AM: 08 August 2023
Accepted: 27 July 2023
Revised: 14 July 2023
Received: 12 October 2022
Published in TOMM Volume 20, Issue 3
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 480Total Downloads
- Downloads (Last 12 months)154
- Downloads (Last 6 weeks)9
Reflects downloads up to 20 Feb 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in