Incorporating concept ontology to enable probabilistic concept reasoning for multi-level image annotation

To enable automatic multi-level image annotation, we have addressed two inter-related important issues:(1)more effective framework for image content representation and feature extraction to characterize the middle-level semantics of image contents;(2)new framework for hierarchical probabilistic image concept reasoning and detection. To address the first issue salient objects are used as the semantic building blocks to characterize the middle-level semantics of image contents effectively while reducing the image analysis cost significantly. We have proposed three approaches to designing the detection functions for automatic salient object detection,and automatic function selection is also supported to find the "right "assumptions of the principal visual properties for the corresponding salient object classes. To address the second issue wehaveproposed a novel framework to incorporate the concept ontology to achieve hierarchical probabilistic image concept reasoning for multi-level image annotation. The concept ontology for a large-scale public image database called Label Me is semi-automatically derived from the available image labels by using WordNet The image concepts at the first level of the concept ontology are used to characterize the most specific semantics of image contents with the smallest variations, and their correspondences with the semantic building blocks (i.e.,salient objects)are well-de fined and can be modeled accurately by using Bayesian networks. In addition,the predictions of the appearances of the higher-level image concepts with large variations are adopted by the underlying concept ontology or by combining the available predictions of the appearances of their children concepts through hierarchical Bayesian networks.Our experiments on a large public dataset have shown that our framework for hierarchical probabilistic image concept reasoning is scalable to diverse image contents (i.e.,large amount of salient object classes)with large within-category variations.