Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Deep-based Self-refined Face-top Coordination

Published: 22 July 2021 Publication History

Abstract

Face-top coordination, which exists in most clothes-fitting scenarios, is challenging due to varieties of attributes, implicit correlations, and tradeoffs between general preferences and individual preferences. We present a Deep-Based Self-Refined (DBSR) system to simulate face-top coordination based on intuition evaluation. To this end, we first establish a well-coordinated face-top (WCFT) dataset from fashion databases and communities. Then, we use a jointly trained CNN Deep Canonical Correlation Analysis (DCCA) method to bridge the semantic face-top gap based on the WCFT dataset to deal with general preferences. Subsequently, an irrelevance-based Optimum-path Forest (OPF) method is developed to adapt the results to individual preferences iteratively. Experimental results and user study demonstrate the effectiveness of our method.

References

[1]
Kaori Abe, Teppei Suzuki, Shunya Ueta, Akio Nakamura, Yutaka Satoh, and Hirokatsu Kataoka. 2017.Changing fashion cultures. arXiv preprint arXiv:1703.07920 (2017).
[2]
Stuti Ajmani, Hiranmay Ghosh, Anupama Mallik, and Santanu Chaudhury. 2013.An ontology based personalized garment recommendation system. In Proceedings of the IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT’13), Vol. 3. IEEE, 17–20.
[3]
Shotaro Akaho. 2006.A kernel method for canonical correlation analysis. arXiv preprint cs/0609071 (2006).
[4]
Ziad Al-Halah, Rainer Stiefelhagen, and Kristen Grauman. 2017.Fashion forward: Forecasting visual style in fashion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, 388–397.
[5]
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013.Deep canonical correlation analysis. In Proceedings of the International Conference on Machine Learning (ICML’13). ACM, 1247–1255.
[6]
Jean-Yves Baudouin and Guy Tiberghien. 2004.Symmetry, averageness, and feature size in the facial attractiveness of women. Acta Psychol. 117, 3 (2004), 313–332.
[7]
Andrea Bottino and Aldo Laurentini. 2010.The analysis of facial beauty: An emerging area of research in pattern analysis. In Proceedings of the International Conference Image Analysis and Recognition. Springer, 425–435.
[8]
Wen Chen, Pipei Huang, Jiaming Xu, Xin Guo, Cheng Guo, Fei Sun, Chao Li, Andreas Pfadler, Huan Zhao, and Binqiang Zhao. 2019.POG: Personalized outfit generation for fashion recommendation at Alibaba iFashion. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2662–2670.
[9]
Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2019.Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 765–774.
[10]
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005.Learning a similarity metric discriminatively, with application to face verification. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1. IEEE, 539–546.
[11]
Andre Tavares Da Silva, Alexandre Xavier Falcão, and Léo Pini Magalhães. 2011.Active learning paradigms for CBIR systems based on optimum-path forest classification. Pattern Recog. 44, 12 (2011), 2971–2978.
[12]
Yael Eisenthal, Gideon Dror, and Eytan Ruppin. 2006.Facial attractiveness: Beauty and the machine. Neural Comput. 18, 1 (2006), 119–142.
[13]
Ruining He, Chunbin Lin, Jianguo Wang, and Julian McAuley. 2016.Sherlock: Sparse hierarchical embeddings for visually-aware one-class collaborative filtering. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI’16). Morgan Kaufmann, 3740–3746.
[14]
Ruining He and Julian McAuley. 2016.VBPR: Visual Bayesian Personalized Ranking from implicit feedback. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. AAAI, 403–410.
[15]
Ruining He, Charles Packer, and Julian McAuley. 2016.Learning compatibility across categories for heterogeneous item recommendation. In Proceedings of the IEEE 16th International Conference on Data Mining (ICDM’16). IEEE, 937–942.
[16]
Tong He and Yang Hu. 2018.FashionNet: Personalized outfit recommendation with deep neural network. arXiv preprint arXiv:1810.02443 (2018).
[17]
Wanjia He, Weiran Wang, and Karen Livescu. 2017.Multi-view recurrent neural acoustic word embeddings. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17).
[18]
Shintami Chusnul Hidayati, Cheng-Chun Hsu, Yu-Ting Chang, Kai-Lung Hua, Jianlong Fu, and Wen-Huang Cheng. 2018.What dress fits me best? Fashion recommendation on the clothing style for personal body shape. In Proceedings of the 26th ACM International Conference on Multimedia (ACM MM’18). ACM, 438–446.
[19]
Harold Hotelling. 1992.Relations between two sets of variates. Biometrika 28, 3/4 (1936), 321–377.
[20]
Min Hou, Le Wu, Enhong Chen, Zhi Li, Vincent W. Zheng, and Qi Liu. 2019. Explainable fashion recommendation: A semantic attribute region guided approach. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19). Macao, 4681–4688.
[21]
Wei-Lin Hsiao and Kristen Grauman. 2017.Learning the latent “look”: Unsupervised discovery of a style-coherent embedding from fashion images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, 4213–4222.
[22]
Wei-Lin Hsiao and Kristen Grauman. 2018.Creating capsule wardrobes from fashion images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, 7161–7170.
[23]
Chia-Wei Hsieh, Chieh-Yun Chen, Chien-Lung Chou, Hong-Han Shuai, Jiaying Liu, and Wen-Huang Cheng. 2019.FashionOn: Semantic-guided image-based virtual try-on with detailed human and clothing information. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). ACM, 275–283.
[24]
Yang Hu, Xi Yi, and Larry S. Davis. 2015.Collaborative fashion recommendation: A functional tensor factorization approach. In Proceedings of the 23rd ACM International Conference on Multimedia (ACM MM’15). ACM, 129–138.
[25]
Tomoharu Iwata, Shinji Wanatabe, and Hiroshi Sawada. 2011.Fashion coordinates recommender system using photographs from fashion magazines. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI’11), Vol. 3. Morgan Kaufmann, 2262–2267.
[26]
Vignesh Jagadeesh, Robinson Piramuthu, Anurag Bhardwaj, Wei Di, and Neel Sundaresan. 2014.Large scale visual recommendations from street fashion images. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1925–1934.
[27]
Wang-Cheng Kang, Eric Kim, Jure Leskovec, Charles Rosenberg, and Julian McAuley. 2019.Complete the look: Scene-based complementary product recommendation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 10532–10541.
[28]
M. Hadi Kiapour, Kota Yamaguchi, Alexander C. Berg, and Tamara L. Berg. 2014.Hipster wars: Discovering elements of fashion styles. In Proceedings of the European Conference on Computer Vision (ECCV’14). Springer, 472–488.
[29]
Dmitry Kornilov. 2019. Recommendation system based on a user’s physical features. US Patent App. 15/826,533.
[30]
Sudhir Kumar and Mithun Das Gupta. 2019. GAN: Complementary fashion item recommendation. arXiv preprint arXiv:1906.05596 (2019).
[31]
Honglin Li, Masahiro Toyoura, Kazumi Shimizu, Wei Yang, and Xiaoyang Mao. 2016.Retrieval of clothing images based on relevance feedback with focus on collar designs. Vis. Comput. 32, 10 (2016), 1351–1363.
[32]
Yuncheng Li, Liangliang Cao, Jiang Zhu, and Jiebo Luo. 2017.Mining fashion outfit composition using an end-to-end deep learning approach on set data. IEEE Trans. Multim. 19, 8 (2017), 1946–1955.
[33]
Lizi Liao, Xiangnan He, Bo Zhao, Chong-Wah Ngo, and Tat-Seng Chua. 2018.Interpretable multimodal retrieval for fashion products. In Proceedings of the 26th ACM International Conference on Multimedia (ACM MM’18). ACM, 1571–1579.
[34]
Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, and Maarten De Rijke. 2020.Explainable outfit recommendation with joint outfit matching and comment generation. IEEE Trans. Knowl. Data Eng. 32, 8 (2020), 1502–1516.
[35]
Jian Liu, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Fuzheng Zhuang, Jiajie Xu, Xiaofang Zhou, and Hui Xiong. 2019.Deep cross networks with aesthetic preference for cross-domain recommendation. arXiv preprint arXiv:1905.13030 (2019).
[36]
Luoqi Liu, Junliang Xing, Si Liu, Hui Xu, Xi Zhou, and Shuicheng Yan. 2014.Wow! you are so beautiful today!ACM Trans. Multim. Comput. Commun. Applic. 11, 1s (2014), 1–20.
[37]
Si Liu, Jiashi Feng, Zheng Song, Tianzhu Zhang, Hanqing Lu, Changsheng Xu, and Shuicheng Yan. 2012.Hi, magic closet, tell me what to wear! In Proceedings of the 20th ACM International Conference on Multimedia (ACM MM’12). ACM, 619–628.
[38]
Zhi Lu, Yang Hu, Yunchao Jiang, Yan Chen, and Bing Zeng. 2019.Learning binary code for personalized fashion recommendation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 10562–10570.
[39]
Yihui Ma, Jia Jia, Suping Zhou, Jingtian Fu, Yejun Liu, and Zijian Tong. 2017.Towards better understanding the clothing fashion styles: A multimodal deep learning approach. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI, 38–44.
[40]
Yunshan Ma, Xun Yang, Lizi Liao, Yixin Cao, and Tat-Seng Chua. 2019.Who, where, and what to wear? extracting fashion knowledge from social media. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). ACM, 257–265.
[41]
Utkarsh Mall, Kevin Matzen, Bharath Hariharan, Noah Snavely, and Kavita Bala. 2019.GeoStyle: Discovering fashion trends and events. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’19). IEEE, 411–420.
[42]
Kevin Matzen, Kavita Bala, and Noah Snavely. 2017.StreetStyle: Exploring world-wide clothing styles from millions of photos. arXiv preprint arXiv:1706.01869 (2017).
[43]
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015.Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 43–52.
[44]
Mahesh Chandra Mukkamala and Matthias Hein. 2017.Variants of RMSProp and Adagrad with logarithmic regret bounds. In Proceedings of the 34th International Conference on Machine Learning (ICML’17), Vol. 70. ACM, 2545–2553.
[45]
Tam V. Nguyen, Si Liu, Bingbing Ni, Jun Tan, Yong Rui, and Shuicheng Yan. 2012.Sense beauty via face, dressing, and/or voice. In Proceedings of the 20th ACM International Conference on Multimedia (ACM MM’12). ACM, 239–248.
[46]
Hosnieh Sattar, Gerard Pons-Moll, and Mario Fritz. 2019.Fashion is taking shape: Understanding clothing preference based on body shape from online sources. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV’19). IEEE, 968–977.
[47]
Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, and Raquel Urtasun. 2015.Neuroaesthetics in fashion: Modeling the perception of fashionability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). IEEE, 869–877.
[48]
Karen Simonyan and Andrew Zisserman. 2014.Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[49]
Guang-Lu Sun, Zhi-Qi Cheng, Xiao Wu, and Qiang Peng. 2018.Personalized clothing recommendation combining user social circle and fashion style consistency. Multim. Tools Applic. 77, 14 (2018), 17731–17754.
[50]
Moeko Takagi, Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2017.What makes a style: Experimental analysis of fashion prediction. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW’17). IEEE, 2247–2253.
[51]
Pongsate Tangseng and Takayuki Okatani. 2020.Toward explainable fashion recommendation. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE, 2153–2162.
[52]
Kristen Vaccaro, Sunaya Shivakumar, Ziqiao Ding, Karrie Karahalios, and Ranjitha Kumar. 2016.The elements of fashion style. In Proceedings of the 29th Symposium on User Interface Software and Technology. 777–785.
[53]
Dario Riccardo Valenzano, Andrea Mennucci, Giandonato Tartarelli, and Alessandro Cellerino. 2006.Shape analysis of female facial attractiveness. Vis. Res. 46, 8–9 (2006), 1282–1291.
[54]
Andreas Veit, Balazs Kovacs, Sean Bell, Julian McAuley, Kavita Bala, and Serge Belongie. 2015.Learning visual clothing style with heterogeneous dyadic co-occurrences. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). IEEE, 4642–4650.
[55]
Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2017.Adversarial cross-modal retrieval. In Proceedings of the 25th ACM International Conference on Multimedia (ACM MM’17). ACM, 154–162.
[56]
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. 2015.On deep multi-view representation learning. In Proceedings of the 32nd International Conference on Machine Learning (ICML’15), Vol. 37. ACM, 1083–1092.
[57]
Zhonghua Wu, Guosheng Lin, Qingyi Tao, and Jianfei Cai. 2019.M2E-Try On Net: Fashion from model to everyone. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). ACM, 293–301.
[58]
Duorui Xie, Lingyu Liang, Lianwen Jin, Jie Xu, and Mengru Li. 2015.SCUT-FBP: A benchmark dataset for facial beauty perception. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 1821–1826.
[59]
Kota Yamaguchi, Tamara L. Berg, and Luis E. Ortiz. 2014.Chic or social: Visual popularity analysis in online fashion networks. In Proceedings of the 22nd ACM International Conference on Multimedia (ACM MM’14). ACM, 773–776.
[60]
Wei Yang, Masahiro Toyoura, and Xiaoyang Mao. 2012.Hairstyle suggestion using statistical learning. In Proceedings of the International Conference on Multimedia Modeling (MMM’12). Springer, 277–287.
[61]
Xun Yang, Xiangnan He, Xiang Wang, Yunshan Ma, Fuli Feng, Meng Wang, and Tat-Seng Chua. 2019.Interpretable fashion matching with rich attributes. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 775–784.
[62]
Xun Yang, Yunshan Ma, Lizi Liao, Meng Wang, and Tat-Seng Chua. 2019.TransNFCM: Translation-based neural fashion compatibility modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. AAAI, 403–410.
[63]
Lap-Fai Yu, Sai Kit Yeung, Demetri Terzopoulos, and Tony F. Chan. 2012.DressUp!: Outfit synthesis through automatic optimization.ACM Trans. Graph. 31, 6 (2012).
[64]
Wenhui Yu, Huidi Zhang, Xiangnan He, Xu Chen, Li Xiong, and Zheng Qin. 2018.Aesthetic-based clothing recommendation. In Proceedings of the World Wide Web Conference (WWW’18). ACM, 649–658.
[65]
Yi Yu, Suhua Tang, Francisco Raposo, and Lei Chen. 2019.Deep cross-modal correlation learning for audio and lyrics in music retrieval. ACM Trans. Multim. Comput. Commun. Applic. 15, 1 (2019), 1–16.
[66]
Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. 2019.Deep supervised cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE, 10386–10395.
[67]
Zhengzhong Zhou, Xiu Di, Wei Zhou, and Liqing Zhang. 2018.Fashion sensitive clothing recommendation using hierarchical collocation model. In Proceedings of the 26th ACM International Conference on Multimedia (ACM MM’18). ACM, 1119–1127.

Cited By

View all
  • (2024)Boosting Semi-Supervised Learning with Dual-Threshold Screening and Similarity LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3672563Online publication date: 12-Jun-2024
  • (2024)MF2ShrT: Multimodal Feature Fusion Using Shared Layered Transformer for Face Anti-spoofingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364081720:6(1-21)Online publication date: 8-Mar-2024
  • (2024)Learning Offset Probability Distribution for Accurate Object DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363721420:5(1-24)Online publication date: 22-Jan-2024
  • Show More Cited By

Index Terms

  1. Deep-based Self-refined Face-top Coordination

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 3
    August 2021
    443 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/3476118
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 July 2021
    Accepted: 01 January 2021
    Revised: 01 January 2021
    Received: 01 April 2020
    Published in TOMM Volume 17, Issue 3

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Fashion analysis
    2. personalized face-top coordination
    3. deep cross-modal learning
    4. canonical correlation analysis
    5. relevance feedback
    6. optimum-path forest

    Qualifiers

    • Research-article
    • Refereed

    Funding Sources

    • National Key R&D Program of China
    • Ningbo Major Special Projects of the “Science and Technology Innovation 2025”
    • National Natural Science Foundation of China

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)13
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 11 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Boosting Semi-Supervised Learning with Dual-Threshold Screening and Similarity LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3672563Online publication date: 12-Jun-2024
    • (2024)MF2ShrT: Multimodal Feature Fusion Using Shared Layered Transformer for Face Anti-spoofingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364081720:6(1-21)Online publication date: 8-Mar-2024
    • (2024)Learning Offset Probability Distribution for Accurate Object DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363721420:5(1-24)Online publication date: 22-Jan-2024
    • (2024)Head3D: Complete 3D Head Generation via Tri-plane Feature DistillationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363571720:6(1-20)Online publication date: 8-Mar-2024
    • (2024)LiteWiSys: A Lightweight System for WiFi-based Dual-task Action PerceptionACM Transactions on Sensor Networks10.1145/363217720:4(1-19)Online publication date: 11-May-2024
    • (2024)Gait Attribute Recognition: A New Benchmark for Learning Richer Attributes From Human Gait PatternsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.331893419(1-14)Online publication date: 1-Jan-2024
    • (2024)Neural network-based cross-channel chroma prediction for versatile video codingThe Journal of Supercomputing10.1007/s11227-023-05868-y80:9(12166-12185)Online publication date: 8-Feb-2024
    • (2023)Video Codec Using Machine Learning Based on Parametric Orthogonal FiltersOptical Memory and Neural Networks10.3103/S1060992X2304002132:4(226-232)Online publication date: 22-Dec-2023
    • (2023)Voice-Face Homogeneity Tells DeepfakeACM Transactions on Multimedia Computing, Communications, and Applications10.1145/362523120:3(1-22)Online publication date: 10-Nov-2023
    • (2023)Contrastive Attention-guided Multi-level Feature Registration for Reference-based Super-resolutionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361649520:2(1-21)Online publication date: 18-Oct-2023
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media