Abstract
With the advent of digital cameras, the number of digital images is on the increase. As a result, image collection summarization systems are proposed to provide users with a condense set of summary images as a representative set to the original high volume image set. In this paper, a semantic knowledge-based approach for image collection summarization is presented. Despite ontology and knowledge-based systems have been applied in other areas of image retrieval and image annotation, most of the current image summarization systems make use of visual or numeric metrics for conducting the summarization. Also, some image summarization systems jointly model visual data of images together with their accompanying textual or social information, while these side data are not available out of the context of web or social images. The main motivation of using ontology approach in this study is its ability to improve the result of computer vision tasks by the additional knowledge which it provides to the system. We defined a set of ontology based features to measure the amount of semantic information contained in each image. A semantic similarity graph was made based on semantic similarities. Summary images were then selected based on graph centrality on the similarity graph. Experimental results showed that the proposed approach worked well and outperformed the current image summarization systems.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Abdollahpour Z, Samani ZR, Moghaddam ME (2015) Image classification using ontology based improved visual words. In: 2015 23rd Iranian conference on electrical engineering, IEEE, pp 694–698
Bannour H, Hudelot C (2014) Building and using fuzzy multimedia ontologies for semantic image annotation. Multimed Tools Appl 72(3):2107–2141
Barrilero M et al (2011) Innetwork content based image recommendation system for Contentaware Networks. In: IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)
Berg TL, Forsyth D (2007) Automatic ranking of iconic images. University of California, Berkeley, Tech Rep
Brin S, Page L (2012) Reprint of: the anatomy of a large-scale hypertextual web search engine. Comput Netw 56(18):3825–3833
Camargo JE, González FA (2016) Multimodal latent topic analysis for image collection summarization. Inf Sci 328:270–287
Chatfield K et al (2011) The devil is in the details: an evaluation of recent feature encoding methods
Clough P, Joho H, Sanderson M (2005) Automatically organising images using concept hierarchies. In: Proceedings of the multimedia workshop running at ACM SIGIR conference. Sheffield.
Crandall DJ et al (2009) Mapping the world’s photos. In: Proceedings of the 18th international conference on world wide web
Das D, Martins AF (2007) A survey on automatic text summarization. Literature survey for the language and statistics II course at CMU 4:192–195
Delest M, Don A, Benois-Pineau J (2006) DAG-based visual interfaces for navigation in indexed video content. Multimed Tools Appl 31(1):51–72
Deng J, Berg AC, Fei-Fei L (2011) Hierarchical semantic indexing for large scale image retrieval. in Computer Vision and Pattern Recognition (CVPR), 2011 I.E. Conference on. IEEE
Deng J et al (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009
Elleuch N, Ammar AB, Alimi AM (2015) A generic framework for semantic video indexing based on visual concepts/contexts detection. Multimed Tools Appl 74(4):1397–1421
Fan J, Gao Y, Luo H (2008) Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. Image Process IEEE Trans 17(3):407–426
Fan J, Yuli G et al (2008) A novel approach to enable semantic and visual image summarization for exploratory image search. In: Proceedings of the 1st ACM international conference on multimedia information retrieval
Fang H et al (2015) Topic aspect-oriented summarization via group selection. Neurocomputing 149:1613–1619
Forsati R, Shamsfard M (2016) Symbiosis of evolutionary and combinatorial ontology mapping approaches. Inf Sci
Gehler P, Nowozin S (2009) On feature combination for multiclass object classification. In: Computer Vision, 2009 I.E. 12th International Conference on. IEEE
Griffin G, Perona P (2008) Learning and using taxonomies for fast visual categorization. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE
Jaffe A et al (2006) Generating summaries and visualization for large collections of geo-referenced photographs. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval
Jeong J-W et al (2012) Visual summarization of the social image collection using image attractiveness learned from social behaviors. Multimedia and expo (ICME), 2012 I.E. International Conference on 538–543
Jia Y et al (2008) Finding image exemplars using fast sparse affinity propagation. In: Proceedings of the 16th ACM international conference on multimedia
Jiao B et al (2012) Visually summarizing web pages through internal and external images. IEEE Trans Multimed 14(6):1673–1683
Jing Y, Baluja S (2008) Visualrank: applying pagerank to large-scale image search. Pattern Anal Mach Intell IEEE Trans 30(11):1877–1890
Jing Y, Baluja S, Rowley H (2007) Canonical image selection from the web. In: Proceedings of the 6th ACM international conference on image and video retrieval
Kennedy LS, Chang S-F, Kozintsev IV (2006) To search or to label?: predicting the performance of search-based automatic image classifiers. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval
Kennedy LS, Naaman M (2008) Generating diverse and representative image search results for landmarks. In: Proceedings of the 17th international conference on world wide web
Khosla A et al (2013) Large-scale video summarization using web-image priors. In: Computer Vision and Pattern Recognition (CVPR), 2013 I.E. conference on. IEEE
Kim G, Sigal L, Xing EP (2014) Joint summarization of large-scale collections of web images and videos for storyline reconstruction. In: Computer Vision and Pattern Recognition (CVPR), 2014 I.E. Conference on. IEEE
Koller, T.G.a.D. Discriminative Learning of Relaxed Hierarchy for Large-scale Visual Recognition. in in Computer Vision (ICCV), 2011 I.E. International Conference on. 2011.
Latha TKK (2015) Optimization of sparse dictionary model for multimodal image summarization using firefly algorithm. Int J Appl Eng Res 10(55):1896–1901
Lemos FD et al (2012) Towards a context-aware photo recommender system. In: Context-aware recommender system workshops
Li C, Feng Z, Han Y (2015) Image attribute learning with ontology guided fused lasso. Multimed Tools Appl 1–15
Li L, Jiang S, Huang Q (2012) Learning hierarchical semantic description via mixed-norm regularization for image understanding. IEEE Trans Multimed 14(5):1401–1413
Li Y, Merialdo B (2010) VERT: automatic evaluation of video summaries. In: Proceedings of the international conference on multimedia
Li M, Zhao C, Tang J (2013) Hybrid image summarization by hypergraph partition. Neurocomputing 119:41–48
Li L-J et al (2010) Building and using a semantivisual image hierarchy. In: Computer Vision and Pattern Recognition (CVPR), 2010 I.E. Conference on. IEEE
Lin C-Y, Hovy E (2003) Automatic evaluation of summaries using n-gram co-occurrence statistics. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics
Lowe, D.G., Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2005: p. 91--110.
Marszałek M, Schmid C (2008) Constructing category hierarchies for visual recognition, in Computer Vision–ECCV 2008. Springer 479–491
Mei S et al (2015) Video summarization via minimum sparse reconstruction. Pattern Recogn 48(2):522–533
Naci SU et al (2008) The COST292 experimental framework for rushes summarization task in TRECVID 2008. In: Proceedings of the 2nd ACM TRECVID video summarization workshop. ACM
Newman M (2010) Networks: an introduction. Oxford University Press, Oxford
Nister D, Stewenius H (2006) Scalable recognition with a vocabulary tree. In: Computer Vision and Pattern Recognition, 2006 I.E. Computer Society Conference on. IEEE
Over P, Smeaton AF, Awad G (2008) The TRECVid 2008 BBC rushes summarization evaluation. In: Proceedings of the 2nd ACM TRECVid video summarization workshop. ACM
Palmer S, Rosch E, Chase P (1981) Canonical perspective and the perception. Atten Perform 4:135–151
Pang Y, Qiang H et al (2011) Summarizing tourist destinations by mining user-generated travelogues and photos. Comput Vis Image Underst 115(3):352–363
Poole D (2014) Linear algebra: a modern introduction. Cengage Learning
Qian G, Sural S, Pramanik S (2002) A comparative analysis of two distance measures in color image databases. In: Image Processing. 2002. Proceedings. 2002 International Conference on. IEEE
Qian G et al (2004) Similarity between Euclidean and cosine angle distance for nearest neighbor queries. In: Proceedings of the 2004 ACM symposium on applied computing. ACM
Raguram R, Lazebnik S (2008) Computing iconic summaries of general visual concepts. In: Computer vision and pattern recognition workshops, 2008. CVPRW’08. IEEE Computer Society Conference on
Rudinac S, Larson M, Hanjalic A (2013) Learning crowdsourced user preferences for visual summarization of image collections. Multimed IEEE Trans 15(6):1231–1243
Russell BC et al (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173
Schadd FC, Roos N (2012) Coupling of wordnet entries for ontology mapping using virtual documents. In: Proceedings of the ISWC
Shen X, Tian X (2015) Multi-modal and multi-scale photo collection summarization. Multimed Tools Appl 1–15
Simon I, Noah S, Seitz SM (2007) Scene summarization for online image collections. In: IEEE 11th International Conference on Computer Vision (ICCV 2007)
Tschiatschek S et al (2014) Learning mixtures of submodular functions for image collection summarization. In: Advances in neural information processing systems
van Leuken RH et al (2009) Visual diversification of image search results. In: Proceedings of the 18th international conference on world wide web
Verma N et al (2012) Learning hierarchical similarity metrics. In: Computer Vision and Pattern Recognition (CVPR), 2012 I.E. Conference on
Wang J, Jia L, Hua X-S (2011) Interactive browsing via diversified visual summarization for image search results. Multimedia Systems 17(5):379–391
Xu H et al (2011) Hybrid image summarization. In: Proceedings of the 19th ACM international conference on multimedia
Yang L, Adviser-Johnstone JK (2011) Mining canonical views from internet image collections
Yang Y, Chen S-C (2012) Disaster image filtering and summarization based on multi-layered affinity propagation. In: IEEE International Symposium on Multimedia (ISM), 2012
Yang YH et al (2008) ContextSeer: context search and recommendation at query time for shared consumer photos. In: Proceedings of the 16th ACM international conference on multimedia
Yang C et al (2013) Image collection summarization via dictionary learning for sparse representation. Pattern Recogn 46:948–961
Yu H et al (2014) A joint optimization model for image summarization based on image content and tags. In: Twenty-eighth AAAI conference on artificial intelligence
Zhang L, Lin F, Zhang B (2001). Support vector machine learning for image retrieval. in Image Processing, 2001. Proceedings. 2001 International Conference on. IEEE
Zhao B, Li F, Xing EP (2011) Large-scale category structure aware image categorization. Adv Neur Inf Process Syst 1251–1259
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Samani, Z.R., Moghaddam, M.E. A knowledge-based semantic approach for image collection summarization. Multimed Tools Appl 76, 11917–11939 (2017). https://doi.org/10.1007/s11042-016-3840-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-016-3840-1