Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

ISRE-Framework: nonlinear and multimodal exploration of image search result spaces

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The extensive information delivery power and an immense volume of image objects make them frequently use multimedia content over the web. However, access to desired image objects to satisfy visual information needs by employing primitive exploration paradigms is difficult. Traditionally, the linear presentation of web image results often leads to reachability and navigation issues. Alternatively, nonlinear approaches provide navigation in web image results. The in-depth browsing to access particular web image results is challenging. In this research, we proposed an exploration framework to browse and explore web image results. We addressed the associated exploration issues, i.e., reachability and navigation in browsing and visualization. The framework enables the nonlinear and multimodal exploration of web image results by representing them in a graph-cluster data model and enabling an interactive exploration mechanism. The graph-cluster data model mainly employs and modifies Zahn’s method and particular algorithms to transform the web image results into specific nonlinear and multimodal search results spaces. The exploration mechanism enables reachability, navigation, browsing, and visualization of web image results in an integrated way. We instantiated the proposed framework over a real dataset of image objects and employed empirical, usability, and comparison tests to evaluate the proposed exploration framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://mathworld.wolfram.com/MaximumSpanningTree.html

  2. https://www.gettyimages.com/

  3. https://sites.google.com/site/dctresearch/Home/content-based-image-retrieval

  4. https://www.imageclef.org/2012/photo-flickr/dataset

  5. https://trecvid.nist.gov/

  6. https://www.imageclef.org/photodata

  7. https://vcl.iti.gr/project/i-search/

  8. https://vcl.iti.gr/dataset/i-search-multimodal-dataset/

  9. https://www.flickr.com/

  10. https://chatzichristofis.info/?page_id=19

  11. https://doi.org/10.5281/zenodo.511229

  12. https://vcl.iti.gr/project/i-search/

  13. https://www.flickr.com/

  14. https://chatzichristofis.info/?page_id=15

  15. https://lucene.apache.org

  16. https://www.elastic.co/

  17. https://www.python.org/

  18. https://www.elastic.co/

  19. https://pypi.org/project/scipy/

  20. https://pypi.org/project/matplotlib/

  21. https://networkx.github.io/

  22. https://js.cytoscape.org/

  23. https://pypi.org/project/networkx/

  24. https://js.cytoscape.org/

  25. https://garyperlman.com/quest/quest.cgi?form=QUIS

References

  1. Ahn J W, Brusilovsky P (2013) Adaptive visualization for exploratory information retrieval. Inf Process Manag 49(5):1139–1164

    Article  Google Scholar 

  2. Ahrens J, Jourdain S, OLeary P, Patchett J, Rogers D H, Petersen M (2014) An image-based approach to extreme scale in situ visualization and analysis. In: SC’14: proceedings of the international conference for high performance computing, networking, storage and analysis. IEEE, pp 424–434

  3. André P, Cutrell E, Tan D S, Smith G (2009) Designing novel image search interfaces by understanding unique characteristics and usage. In: IFIP conference on human-computer interaction. Springer, pp 340–353

  4. Axenopoulos A, Daras P, Malassiotis S, Croce V, Lazzaro M, Etzold J, Grimm P, Massari A, Camurri A, Steiner T et al (2012) I-search: a unified framework for multimodal search and retrieval. In: The future internet assembly. Springer, pp 130–141

  5. Baeza-Yates R, Ribeiro-Neto B, et al. (1999) Modern information retrieval, vol 463. ACM Press, New York

    Google Scholar 

  6. Chagas G O, Lorena L A N, dos Santos RDC (2019) A hybrid heuristic for the overlapping cluster editing problem. Appl Soft Comput 81:105482

    Article  Google Scholar 

  7. Chatzichristofis S A, Boutalis YS (2008) CEDD: Color and edge directivity descriptor: A compact descriptor for image indexing and retrieval. In: International conference on computer vision systems. Springer, pp 312–322

  8. Chatzichristofis S A, Zagoris K, Boutalis Y S, Papamarkos N (2010) Accurate image retrieval based on compact composite descriptors and relevance feedback information. Int J Pattern Recognit Artif Intell 24(02):207–244

    Article  Google Scholar 

  9. Chaudhary C, Goyal P, Tuli S, Banthia S, Goyal N, Chen Y P P (2019) A novel multimodal clustering framework for images with diverse associated text. Multimed Tools Appl 78(13):17623–17652

    Article  Google Scholar 

  10. Chen J, Lu J (2019) A Clustering Algorithm Based on Minimum Spanning Tree and Density. In: 2019 IEEE 4th international conference on big data analytics (ICBDA). IEEE, pp 1–4

  11. Chin J P, Diehl V A, Norman KL (1988) Development of an instrument measuring user satisfaction of the human-computer interface. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 213–218

  12. Dimitrov D, Lemmerich F, Flöck F, Strohmaier M (2018) Query for architecture, click through military: Comparing the roles of search and navigation on Wikipedia. In: Proceedings of the 10th ACM conference on web science, pp 371–380

  13. dos Santos Belo L, Caetano C A Jr, do Patrocínio ZK G Jr, Guimarães SJF (2016) Summarizing video sequence using a graph-based hierarchical approach. Neurocomputing 173:1001–1016

    Article  Google Scholar 

  14. Duygulu P, Barnard K, de Freitas J F, Forsyth DA (2002) Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In: European conference on computer vision. Springer, pp 97–112

  15. Garces E, Agarwala A, Hertzmann A, Gutierrez D (2017) Style-based exploration of illustration datasets. Multimed Tools Appl 76 (11):13067–13086

    Article  Google Scholar 

  16. Gormley C, Tong Z (2015) Elasticsearch: the definitive guide: a distributed real-time search and analytics engine. O’Reilly Media Inc.

  17. Grubinger M, Clough P, Müller H, Deselaers T (2006) Overview of the ImageCLEF 2006 photographic retrieval and object annotation tasks. In Workshop of the Cross-Language Evaluation Forum for European Languages Springer, Berlin, Heidelberg, vol 2, pp 579–594

  18. Hay L, Duffy A, Grealy M, Tahsiri M, McTeague C, Vuletic T (2020) A novel systematic approach for analysing exploratory design ideation. J Eng Des 127–149

  19. Hearst M A (2006) Clustering versus faceted categories for information exploration. Commun ACM 49(4):59–61

    Article  Google Scholar 

  20. Hearst M (2009) Search user interfaces. Cambridge University Press, New York

    Book  Google Scholar 

  21. Hoque E, Hoeber O, Gong M (2011) Evaluating the trade-offs between diversity and precision for Web image search using concept-based query expansion. In: 2011 IEEE/WIC/ACM international conferences on web intelligence and intelligent agent technology, vol 3. IEEE, pp 130–133

  22. Huiskes M J, Lew MS (2008) The mir flickr retrieval evaluation. In: Proceedings of the 1st ACM international conference on multimedia information retrieval, pp 39–43

  23. Jain A, Nandakumar K, Ross A (2005) Score normalization in multimodal biometric systems. Pattern recognition 38(12):2270-2285

    Article  Google Scholar 

  24. Käki M (2005) Findex: search result categories help users when document ranking fails. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 131–140

  25. Kennedy L S, Naaman M (2008) Generating diverse and representative image search results for landmarks. In: Proceedings of the 17th international conference on World Wide Web, pp 297–306

  26. Kherfi ML (2008) Advances in human-computer interaction. I-Tech Education and Publishing KG, Vienna, pp 215–240

    Google Scholar 

  27. Kogge P M (2016) Jaccard coefficients as a potential graph benchmark. In: 2016 IEEE international parallel and distributed processing symposium workshops (IPDPSW). IEEE, 921–928

  28. Lewis EC (2011) Image representation and interactivity: an exploration of utility values information-needs and image interactivity. ERIC

  29. Lu S, Mei T, Wang J, Zhang J, Wang Z, Li S (2014) Cache design of ssd-based search engine architectures: An experimental study. ACM Trans Inf Syst (TOIS) 32(4):1–26

    Article  Google Scholar 

  30. Ma KL (1999) Image graphs-a novel approach to visual data exploration. IEEE

  31. Marchionini G (2006) Exploratory search: from finding to understanding. Commun ACM 49(4):41–46

    Article  Google Scholar 

  32. Mourchid Y, El Hassouni M, Cherifi H (2019) A general framework for complex network-based image segmentation. Multimed Tools Appl 78 (14):20191–20216

    Article  Google Scholar 

  33. Mu C, Zhao J, Yang G, Zhang J, Yan Z (2018) Towards practical visual search engine within elasticsearch. arXiv:1806.08896

  34. Palagi E, Gandon F, Giboin A, Troncy R (2017) A survey of definitions and models of exploratory search. In: Proceedings of the 2017 ACM workshop on exploratory search and interactive data analytics, pp 3–8

  35. Park J Y, O’Hare N, Schifanella R, Jaimes A, Chung CW (2015) A large-scale study of user image search behavior on the web. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp 985–994

  36. Petkos G, Schinas M, Papadopoulos S, Kompatsiaris Y (2017) Graph-based multimodal clustering for social multimedia. Multimed Tools Appl 76 (6):7897–7919

    Article  Google Scholar 

  37. Pienta R, Kahng M, Lin Z, Vreeken J, Talukdar P, Abello J, Parameswaran G, Chau DH (2017) Facets: Adaptive local exploration of large graphs. In: Proceedings of the 2017 SIAM international conference on data mining. SIAM, pp 597–605

  38. Pinto-Cáceres S M, Almeida J, Baranauskas M C C, Torres RDS (2015) Fisir: a flexible framework for interactive search in image retrieval systems. In: International conference on multimedia modeling. Springer, pp 335–347

  39. Rafailidis D, Manolopoulou S, Daras P (2013) A unified framework for multimodal retrieval. Pattern Recogn 46(12):3358–3370

    Article  Google Scholar 

  40. Rashid U, Bhatti M A (2017) A framework to explore results in multiple media information aggregated search. Multimed Tools Appl 76(24):25787–25826

    Article  Google Scholar 

  41. Rashid U, Niaz I A, Bhatti MA (2010) Fusion of multimedia document intra-modality relevancies using linear combination model. In: Advanced techniques in computing sciences and software engineering. Springer, pp 575–580

  42. Rashid U, Viviani M, Pasi G (2016) A graph-based approach for visualizing and exploring a multimedia search result space. Inf Sci 370:303–322

    Article  Google Scholar 

  43. Rashid U, Viviani M, Pasi G, Bhatti MA (2016). In: Flexible query answering systems 2015. Springer, pp 271–282

  44. Richards M (2015) Software architecture patterns, vol 4. O’Reilly Media, Sebastopol

    Google Scholar 

  45. Richter F, Romberg S, Hörster E, Lienhart R (2010) Multimodal ranking for image search on community databases. In: Proceedings of the international conference on multimedia information retrieval, pp 63–72

  46. Sabetghadam S, Lupu M, Bierig R, Rauber A (2015) Reachability analysis of graph modelled collections. In: European conference on information retrieval. Springer, pp 370–381

  47. Sabetghadam S, Lupu M, Bierig R, Rauber A (2018) A faceted approach to reachability analysis of graph modelled collections. Int J Multimed Inf Retr 7(3):157–171

    Article  Google Scholar 

  48. Saddal M, Rashid U, Khattak AS (2019) A browsing approach to explore web image search results. In: 2019 22nd international multitopic conference (INMIC). IEEE, pp 1–6

  49. Saglam A, Baykan N A (2017) Sequential image segmentation based on minimum spanning tree representation. Pattern Recogn Lett 87:155–162

    Article  Google Scholar 

  50. Sarrafzadeh B, Lank E (2017) Improving exploratory search experience through hierarchical knowledge graphs. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pp 145–154

  51. Schinas M, Papadopoulos S, Kompatsiaris Y, Mitkas P A (2016) Mgraph: multimodal event summarization in social media using topic models and graphbased ranking. Int J Multimed Inf Retr 5(1):51–69

    Article  Google Scholar 

  52. Sciascio C D, Sabol V, Veas E (2017) Supporting exploratory search with a visual user-driven approach. ACM Trans Interact Intell Syst (TiiS) 7 (4):1–35

    Article  Google Scholar 

  53. Shrivastav S, Kumar S, Kumar K (2017) Towards an ontology based framework for searching multimedia contents on the web. Multimed Tools Appl 76 (18):18657–18686

    Article  Google Scholar 

  54. Smeaton AF (2005) Large scale evaluations of multimedia information retrieval: The TRECVid experience. In: International conference on image and video retrieval. Springer, pp 11–17

  55. Tullis T S, Stetson JN (2004) A comparison of questionnaires for assessing website usability. In: Usability professional association conference, vol 1, Minneapolis pp. 1–12

  56. Van Zwol R, Sigurbjornsson B, Adapala R, Garcia Pueyo L, Katiyar A, Kurapati K, Muralidharan M, Muthu S, Murdock V, Ng P et al (2010) Faceted exploration of image search results. In: Proceedings of the 19th international conference on World Wide Web, pp 961–970

  57. Wang M, Li H, Tao D, Lu K, Wu X (2012) Multimodal graph-based reranking for web image search. IEEE Trans Image Process 21(11):4649–4661

    Article  MathSciNet  Google Scholar 

  58. White R W, Roth R A (2009) Exploratory search: Beyond the query-response paradigm. Synthesis lectures on information concepts, retrieval, and services 1(1):1–98

  59. Wilson T D (2009) Review of: Morville, Peter and Callender, Jeffrey search patterns: discovery for design

  60. Xie X, Mao J, Liu Y, de Rijke M, Shao Y, Ye Z, Zhang M, Ma S (2019) Grid-based evaluation metrics for web image search. In: The World Wide Web conference, pp 2103–2114

  61. Zahn C T (1971) Graph-theoretical methods for detecting and describing gestalt clusters. IEEE Trans Comput 100(1):68–86

    Article  Google Scholar 

  62. Zhu L, Shen J, Jin H, Zheng R, Xie L (2015) Content-based visual landmark search via multimodal hypergraph learning. IEEE Trans Cybern 45(12):2756–2769

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the provision of the research facilities provided by the Department of Computer Sciences, Quaid-i-Azam University, Islamabad, Pakistan, to conduct this research study.

Author information

Authors and Affiliations

  1. Computer Science, Quaid-i-Azam University, Islamabad, Pakistan

    Maha Saddal, Umer Rashid, Akmal Saeed Khattak & Ghazanfar Farooq

Authors
  1. Maha Saddal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Umer Rashid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Akmal Saeed Khattak
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ghazanfar Farooq
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Umer Rashid.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saddal, M., Rashid, U., Khattak, A.S. et al. ISRE-Framework: nonlinear and multimodal exploration of image search result spaces. Multimed Tools Appl 81, 27275–27308 (2022). https://doi.org/10.1007/s11042-022-12561-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12561-4

Keywords

Search

Navigation