Abstract
Clicking pictures using mobile cameras is an ubiquitous part of daily human lives. However, finding the same photo, after a couple of weeks, is a herculean task. Conventional gallery search is based on a 1D query input, limited to keywords, without any smart understanding of the user’s needs. The user is needed to scroll through hundreds of images to find the one. In this work, we propose a multi-modal canvas based content retrieval system. Our system allows the users to write/draw on a 2-D canvas, instead of a single line. We use the relative position of the keywords to better understand the user’s needs and retrieve the photo he needs faster. Complex ideas of perception, nearness and object relations are more easily explained through a 2-D input, rather than complex sentences. We integrate a handwriting recognition system with a clustering based sketch recognition engine, to allow users a truly multi-modal system to interact with their devices. Our system also allows users to specify the finer features of the objects, thus providing increased ease of retrieval. We show that using the proposed system, users are able to find the required image faster with a \(33\%\) reduction in human effort. Proposed system allows users to creatively search for images rather than monotonously scrolling through hundreds of images.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Song, J., Yu, Q., Song, Y.Z., Xiang, T., Hospedales, T.M.: Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In: IEEE International Conference on Computer Vision (2017)
Pang, K., et al.: Generalising fine-grained sketch-based image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 677–686 (2019)
Zheng, L., Wang, S., Liu, Z., Tian, Q.: Packing and padding: coupled multi-index for accurate image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1939–1946 (2014)
Chatzichristofis, S.A., Boutalis, Y.S.: Fcth: fuzzy color and texture histogram-a low level feature for accurate image retrieval. In: 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services, pp. 191–196 (2008)
Parcalabescu, L., Frank, A.: Exploring phrase grounding without training: contextualisation and extension to text-based image retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 962–963 (2020)
Krishnan, A., Rajesh, S., Shylaja, S.S.: Text-based image retrieval using captioning. In: 2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT), pp. 1–5 (2021)
Collomosse, J., Bui, T., Jin, H.: Livesketch: query perturbations for guided sketch-based visual search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2887 (2019)
Dey, S., Riba, P., Dutta, A., Llados, J., Song, Y.Z.: Doodle to search: practical zero-shot sketch-based image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
Deng, C., Xu, X., Wang, H., Yang, M., Tao, D.: Progressive cross-modal semantic network for zero-shot sketch-based image retrieval. IEEE Trans. Image Process. 29, 8892–8902 (2020)
Xu, X., Yang, M., Yang, Y., Wang, H.: Progressive domain-independent feature decomposition network for zero-shot sketch-based image retrieval (2020)
Bahlmann, C., Burkhardt, H.: The writer independent online handwriting recognition system frog on hand and cluster generative statistical dynamic time warping. IEEE Trans. Pattern Anal. Mach. Intell. 26, 299–310 (2004)
Khomenko, V., Volkoviy, A., Degtyarenko, I., Radyvonenko, O.: Handwriting text/non-text classification on mobile device. In: Fourth International Conference on Artificial Intelligence and Pattern Recognition (AIPR), pp. 42–49 (2017)
Graves, A., Liwicki, M.: A novel connectionist system for unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31, 855–868 (2008)
Sahu, P.P., et al.: Personalized hand writing recognition using continued LSTM training. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1. IEEE (2017)
Carbune, V., et al.: Fast multi-language LSTM-based online handwriting recognition. Int. J. Doc. Anal. Recogn. (IJDAR) 23, 1–14 (2020)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Ha, D., Eck, D.: A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477 (2017)
Kuznetsova, A., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982 (2018)
Sun, Z., Wang, C., Zhang, L., Zhang, L.: Free hand-drawn sketch segmentation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 626–639. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33718-5_45
Liwicki, M., Bunke, H.: IAM-OnDB-an on-line English sentence database acquired from handwritten text on a whiteboard. In: International Conference on Document Analysis and Recognition, pp. 956–961 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sahu, P.P., Singh, V., Veera, V. (2023). Multi Modal 2-D Canvas Based Gallery Content Retrieval. In: Gupta, D., Bhurchandi, K., Murala, S., Raman, B., Kumar, S. (eds) Computer Vision and Image Processing. CVIP 2022. Communications in Computer and Information Science, vol 1776. Springer, Cham. https://doi.org/10.1007/978-3-031-31407-0_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-31407-0_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31406-3
Online ISBN: 978-3-031-31407-0
eBook Packages: Computer ScienceComputer Science (R0)