research-article

Circle & Search: Attribute-Aware Shoe Retrieval

Authors:

Shuicheng YanAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 11, Issue 1

Article No.: 3, Pages 1 - 21

https://doi.org/10.1145/2632165

Published: 04 September 2014 Publication History

Abstract

Taking the shoe as a concrete example, we present an innovative product retrieval system that leverages object detection and retrieval techniques to support a brand-new online shopping experience in this article. The system, called Circle & Search, enables users to naturally indicate any preferred product by simply circling the product in images as the visual query, and then returns visually and semantically similar products to the users. The system is characterized by introducing attributes in both the detection and retrieval of the shoe. Specifically, we first develop an attribute-aware part-based shoe detection model. By maintaining the consistency between shoe parts and attributes, this shoe detector has the ability to model high-order relations between parts and thus the detection performance can be enhanced. Meanwhile, the attributes of this detected shoe can also be predicted as the semantic relations between parts. Based on the result of shoe detection, the system ranks all the shoes in the repository using an attribute refinement retrieval model that takes advantage of query-specific information and attribute correlation to provide an accurate and robust shoe retrieval. To evaluate this retrieval system, we build a large dataset with 17,151 shoe images, in which each shoe is annotated with 10 shoe attributes e.g., heel height, heel shape, sole shape, etc.). According to the experimental result and the user study, our Circle & Search system achieves promising shoe retrieval performance and thus significantly improves the users' online shopping experience.

References

[1]

Relja Arandjelovic and Andrew Zisserman. 2011. Smooth object retrieval using a bag of boundaries. In Proceedings of the International Conference on Computer Vision (ICCV'11). IEEE, 375--382.

Digital Library

[2]

Tamara L. Berg, Alexander C. Berg, and Jonathan Shih. 2010. Automatic attribute discovery and characterization from noisy web data. In Proceedings of the European Conference on Computer Vision (ECCV'10). Springer, 663--676.

Digital Library

[3]

Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. 2011. Describing people: A poselet-based approach to attribute classification. In Proceedings of the International Conference on Computer Vision (ICCV'11). IEEE, 1543--1550.

Digital Library

[4]

Huizhong Chen, Andrew Gallagher, and Bernd Girod. 2012. Describing clothing by semantic attributes. In Proceedings of the European Conference on Computer Vision (ECCV'12). Springer, 609--623.

Digital Library

[5]

Navneet Dalal and Bill Triggs. 2005. INRIA person dataset. http://pascal.inrialpes.fr/data/human.

[6]

Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09). 1778--1785.

[7]

Pedro Felzenszwalb and Daniel Huttenlocher. 2004. Distance transforms of sampled functions. Tech. rep., Department of Computing and Information Science, Cornell. http://www.cs.cornell.edu/~dph/papers/dt.pdf.

[8]

Pedro Felzenszwalb, Ross B. Girshick, David Mcallester, and Deva Ramanan. 2010. Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intel. 32, 9, 1627--1645.

Digital Library

[9]

Vittorio Ferrari and Andrew Zisserman. 2008. Learning visual attributes. In Proceedings of the Neural Information Processing Systems Conference (NIPS'08).

[10]

Junfeng He, Jinyuan Feng, Xianglong Liu, Tao Cheng, Tai-Hsu Lin, Hyunjin Chung, and Shih-Fu Chang. 2012. Mobile product search with bag of hash bits and boundary reranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 3005--3012.

Digital Library

[11]

Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2011. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intel. 33, 1, 117--128.

Digital Library

[12]

Hongwen Kang, Martial Hebert, Alexei A. Efros, and Takeo Kanade. 2012. Connecting missing links: Object discovery from sparse observations using 5 million product images. In Proceedings of the European Conference on Computer Vision (ECCV'12). Springer, 794--807.

Digital Library

[13]

Adriana Kovashka, Devi Parikh, and Kristen Grauman. 2012. WhittleSearch: Image search with relative attribute feedback. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 2973--2980.

Digital Library

[14]

Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. 2009. Attribute and simile classifiers for face verification. In Proceedings of the International Conference on Computer Vision (ICCV'09). IEEE, 365--372.

[15]

Si Liu, Zheng Song, Guangcan Liu, Changsheng Xu, Hanqing Lu, and Shuicheng Yan. 2012. Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 3330--3337.

Digital Library

[16]

Shiyang Lu, Tao Mei, Jingdong Wang, Jian Zhang, Zhiyong Wang, David Dagan Feng, Jian-Tao Sun, and Shipeng Li. 2012. Browse-to-search. In Proceedings of the 20^th ACM International Conference on Multimedia (ACM-MM'12). ACM Press, New York, 1323--1324.

Digital Library

[17]

Devi Parikh and Kristen Grauman. 2011. Interactively building a discriminative vocabulary of nameable attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 1681--1688.

Digital Library

[18]

Xiaohui Shen, Zhe Lin, Jonathan Brandt, and Ying Wu. 2012. Mobile product image search by automatic query object extraction. In Proceedings of the European Conference on Computer Vision (ECCV'12). Springer, 114--127.

Digital Library

[19]

Behjat Siddiquie, Rogerio Schmidt Feris, and Larry S. Davis. 2011. Image ranking and retrieval based on multi-attribute queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 801--808.

Digital Library

[20]

Yang Wang and Greg Mori. 2010. A discriminative latent model of object classes and attributes. In Proceedings of the European Conference on Computer Vision (ECCV'10). Springer, 155--168.

Digital Library

[21]

Yi Yang and Deva Ramanan. 2011. Articulated pose estimation with flexible mixtures-of-parts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 1385--1392.

Digital Library

Cited By

Morillo PGarcía-García IOrduña JFernández MJuan M(2019)Comparative study of AR versus video tutorials for minor maintenance operationsMultimedia Tools and Applications10.1007/s11042-019-08437-979:11-12(7073-7100)Online publication date: 19-Dec-2019
https://doi.org/10.1007/s11042-019-08437-9
Wang HZhai JTang XYu BMa XChen WGunawi HReed B(2018)SpindleProceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference10.5555/3277355.3277410(561-573)Online publication date: 11-Jul-2018
https://dl.acm.org/doi/10.5555/3277355.3277410
Kyaw ZQi SGao KZhang HZhang LXiao JWang XChua T(2017)Matryoshka PeekIEEE Transactions on Multimedia10.1109/TMM.2017.265542219:6(1272-1284)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1109/TMM.2017.2655422
Show More Cited By

Index Terms

Circle & Search: Attribute-Aware Shoe Retrieval
1. Computing methodologies
  1. Machine learning
    1. Learning settings
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction

Recommendations

DeepShoe: An improved Multi-Task View-invariant CNN for street-to-shop shoe retrieval
Abstract
The difficulty of describing a shoe item seeing on street with text for online shopping demands an image-based retrieval solution. We call this problem street-to-shop shoe retrieval, whose goal is to find exactly the same shoe in the ...
Possibility of guiding arm movement in circle drawing
SMC'09: Proceedings of the 2009 IEEE international conference on Systems, Man and Cybernetics

We tried to guide human action using galvanic vestibular stimulation (GVS). GVS has a possibility of human behavior guidance without any attention. We tried to guide the trajectory of the subjects' hands when as the continuously drew circles. Previous ...
Detecting heel strikes for gait analysis through acceleration flow

In some forms of gait analysis, it is important to be able to capture when the heel strikes occur. In addition, in terms of video analysis of gait, it is important to be able to localise the heel where it strikes on the floor. In this study, a new motion ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 11, Issue 1

August 2014

151 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/2665935

Editor:
Ralf Steinmetz
Technische Universität Darmstadt, Germany

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 September 2014

Accepted: 01 April 2014

Revised: 01 April 2014

Received: 01 August 2013

Published in TOMM Volume 11, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
405
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Morillo PGarcía-García IOrduña JFernández MJuan M(2019)Comparative study of AR versus video tutorials for minor maintenance operationsMultimedia Tools and Applications10.1007/s11042-019-08437-979:11-12(7073-7100)Online publication date: 19-Dec-2019
https://doi.org/10.1007/s11042-019-08437-9
Wang HZhai JTang XYu BMa XChen WGunawi HReed B(2018)SpindleProceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference10.5555/3277355.3277410(561-573)Online publication date: 11-Jul-2018
https://dl.acm.org/doi/10.5555/3277355.3277410
Kyaw ZQi SGao KZhang HZhang LXiao JWang XChua T(2017)Matryoshka PeekIEEE Transactions on Multimedia10.1109/TMM.2017.265542219:6(1272-1284)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1109/TMM.2017.2655422
Zhan HShi BKot A(2017)Cross-Domain Shoe Retrieval With a Semantic Hierarchy of Attribute Classification NetworkIEEE Transactions on Image Processing10.1109/TIP.2017.273634626:12(5867-5881)Online publication date: Dec-2017
https://doi.org/10.1109/TIP.2017.2736346
Cao CDu YAi H(2016)Bag Detection and Retrieval in Street ShotsProceedings, Part I, of the 22nd International Conference on MultiMedia Modeling - Volume 951610.1007/978-3-319-27671-7_65(780-792)Online publication date: 4-Jan-2016
https://dl.acm.org/doi/10.1007/978-3-319-27671-7_65
Zhan HLi SKot A(2015)Tagging the shoe images by semantic attributes2015 IEEE International Conference on Digital Signal Processing (DSP)10.1109/ICDSP.2015.7252005(892-895)Online publication date: Jul-2015
https://doi.org/10.1109/ICDSP.2015.7252005
Huang JFeris RChen QYan S(2015)Cross-Domain Image Retrieval with a Dual Attribute-Aware Ranking NetworkProceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)10.1109/ICCV.2015.127(1062-1070)Online publication date: 7-Dec-2015
https://dl.acm.org/doi/10.1109/ICCV.2015.127
Tao RSmeulders AChang S(2015)Attributes and categories for generic instance search from one example2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2015.7298613(177-186)Online publication date: Jun-2015
https://doi.org/10.1109/CVPR.2015.7298613

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents