Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Triplet Networks Feature Masking for Sketch-Based Image Retrieval

  • Conference paper
  • First Online:
Image Analysis and Recognition (ICIAR 2017)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10317))

Included in the following conference series:

Abstract

Freehand sketches are an intuitive tool for communication and suitable for various applications. In this paper, we present an effective approach that combines triplet networks and an attention mechanism for sketch-based image retrieval (SBIR). The study conducted in this work is based on features extracted using deep convolutional neural networks (ConvNets). In order to overcome the SBIR cross-domain challenge (i.e. searching for photographs from sketch queries), we use triplet loss to train ConvNets to compute shared embedding for both sketches and images. Our main novel contribution is to combine such triplet networks with an attention mechanism. Our approach outperform previous state-of-the-art on challenging SBIR benchmarks. We achieved a recall of 41.66% (at \(k=1\)) for the sketchy database (more than 4% improvement), a Kendal score of 42.9\(\mathcal {T}_\mathrm {b}\) on the TU-Berlin SBIR benchmark (close to 5.5\(\mathcal {T}_\mathrm {b}\) improvement) and a mean average precision (MAP) of 31% on Flickr15k (a category level SBIR benchmark).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Similar content being viewed by others

References

  1. Eitz, M., Hildebrand, K., Boubekeur, T., et al.: Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Vis. Comput. Graph. 17(11), 1624–1636 (2011)

    Article  Google Scholar 

  2. Eitz, M., Hildebrand, K., Boubekeur, T., et al.: An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Comput. Graph. 34(5), 482–498 (2010)

    Article  Google Scholar 

  3. Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117(7), 790–806 (2013)

    Article  Google Scholar 

  4. Collobert, R., Kavukcuoglu, K., Farabet, C.: A matlab-like environment for machine learning. In: BigLearn, NIPS Workshop (No.EPFL-CONF-192376)

    Google Scholar 

  5. Seddati, O., Dupont, S., Mahmoudi, S.: Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: 2015 13th International Workshop on Content-Based Multimedia Indexing (CBMI), pp. 1–6. IEEE (2015)

    Google Scholar 

  6. Sangkloy, P., Burnell, N., Ham, C., et al.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph. (TOG) 35(4), 119 (2016)

    Article  Google Scholar 

  7. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)

  8. Seddati, O., Dupont, S., Mahmoudi, S.: DeepSketch 2: deep convolutional neural networks for partial sketch recognition. In: 2016 14th International Workshop on. IEEE Content-Based Multimedia Indexing (CBMI), pp. 1–6 (2016)

    Google Scholar 

  9. Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 815–823 (2015)

    Google Scholar 

  10. Sznt, B., Pozsegovics, P., Vmossy, Z., et al.: Sketch4matchContent-based image retrieval system using sketches. In: 2011 IEEE 9th International Symposium on Applied Machine Intelligence and Informatics (SAMI), pp. 183–188. IEEE (2011)

    Google Scholar 

  11. Wang, Y., Yu, M., Jia, Q., et al.: Query by sketch: an asymmetric sketch-vs-image retrieval system. In: 2011 4th International Congress on Image and Signal Processing (CISP), pp. 1368–1372. IEEE (2011)

    Google Scholar 

  12. Jin, C., Wang, Z., Zhang, T., et al.: A novel visual-region-descriptor-based approach to sketch-based image retrieval. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 267–274. ACM (2015)

    Google Scholar 

  13. Seddati, O., Dupont, S., Mahmoudi, S.: DeepSketch2Image: deep convolutional neural networks for partial sketch recognition and image retrieval. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 739–741. ACM (2016)

    Google Scholar 

  14. Bozas, K., Izquierdo, E.: Large scale sketch based image retrieval using patch hashing. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Fowlkes, C., Wang, S., Choi, M.-H., Mantler, S., Schulze, J., Acevedo, D., Mueller, K., Papka, M. (eds.) ISVC 2012. LNCS, vol. 7431, pp. 210–219. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33179-4_21

    Chapter  Google Scholar 

  15. Bhattacharjee, S.D., Yuan, J., Hong, W., et al.: Query adaptive instance search using object sketches. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 1306–1315. ACM (2016)

    Google Scholar 

  16. Bui, T., Ribeiro, L., Ponti, M., et al.: Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search. arXiv preprint arXiv:1611.05301 (2016)

  17. Deore, A., Gunjal, B.L.: Advanced sketch based image retrieval system using object boundary selection algorithm (2016)

    Google Scholar 

  18. Li, Q., Han, Y., Dang, J.: Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals. Multimed. Tools Appl. 75(5), 2419–2434 (2016)

    Article  Google Scholar 

  19. Qi, Y., Song, Y-Z., Zhang, H., et al.: Sketch-based image retrieval via Siamese convolutional neural network. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2460–2464. IEEE (2016)

    Google Scholar 

  20. Niu, J., Ma, J., Lu, J., Liu, X., Zhu, Z.: M-SBIR: an improved sketch-based image retrieval method using visual word mapping. In: Amsaleg, L., Guðmundsson, G., Gurrin, C., Jónsson, B., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10133. Springer, Cham (2017). doi:10.1007/978-3-319-51814-5_22

    Chapter  Google Scholar 

  21. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509522 (2002)

    Article  Google Scholar 

  22. Dalal, N.,Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the CVPR, vol. 1, pp. 886–893 (2005)

    Google Scholar 

  23. Qi, Y., Song, Y.-Z., Xiang, T., Zhang, H., Hospedales, T., Li, Y., Guo, J.: Making better use of edges via perceptual grouping. In: Proceedings of the CVPR (2015)

    Google Scholar 

  24. Bui, T., Collomosse, J.: Scalable sketch-based image retrievalusing color gradient features. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, p. 18 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science - TCTS Lab, Université de Mons, Mons, Belgium

    Omar Seddati, Stéphane Dupont & Saïd Mahmoudi

Authors
  1. Omar Seddati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stéphane Dupont
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saïd Mahmoudi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Omar Seddati .

Editor information

Editors and Affiliations

  1. University of Waterloo, Waterloo, ON, Canada

    Fakhri Karray

  2. University of Porto, Porto, Portugal

    Aurélio Campilho

  3. Politechnique Montreal, Montreal, QC, Canada

    Farida Cheriet

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Seddati, O., Dupont, S., Mahmoudi, S. (2017). Triplet Networks Feature Masking for Sketch-Based Image Retrieval. In: Karray, F., Campilho, A., Cheriet, F. (eds) Image Analysis and Recognition. ICIAR 2017. Lecture Notes in Computer Science(), vol 10317. Springer, Cham. https://doi.org/10.1007/978-3-319-59876-5_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-59876-5_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-59875-8

  • Online ISBN: 978-3-319-59876-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation