Abstract
Recent deep learning-based methods for saliency detection have proved the effectiveness of integrating features with different scales. They usually design various complex architectures of network, e.g., multiple networks, to explore the multi-scale information of images, which is expensive in computation and memory. Feature maps produced with different subsampling convolutional layers have different spatial resolutions; therefore, they can be used as the multi-scale features to reduce the costs. In this paper, by exploiting the in-network feature hierarchy of convolutional networks, we propose a novel multi-scale network for saliency detection (MSNSD) consisting of three modules, i.e., bottom-up feature extraction, top-down feature connection and multi-scale saliency prediction. Moreover, to further boost the performance of MSNSD, an input image-aware saliency aggregation method is proposed based on the ridge regression, which combines MSNSD with some well-performed handcrafted shallow models. Extensive experiments on several benchmarks show that the proposed MSNSD outperforms the state-of-the-art saliency methods with less computational and memory complexity. Meanwhile, our aggregation method for saliency detection is effective and efficient to combine deep and shallow models and make them complementary to each other.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Yang, C., Pu, J., Dong, Y., Xie, G.-S., Si, Y., Liu, Z.: Scene classification-oriented saliency detection via the modularized prescription. Vis. Comput. 35, 1–16 (2018)
Zhou, X., Wang, Y., Zhu, Q., Xiao, C., Lu, X.: Ssg: superpixel segmentation and grabcut-based salient object segmentation. Vis. Comput. 35, 1–14 (2018)
Wei, Y., Liang, X., Chen, Y., Shen, X., Cheng, M.-M., Feng, J., Zhao, Y., Yan, S.: Stc: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2314–2320 (2016)
Guo, J., Ren, T., Huang, L., Liu, X., Cheng, M.-M., Wu, G.: Video salient object detection via cross-frame cellular automata. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 325–330. IEEE (2017)
Cheng, M.-M., Zhang, F.-L., Mitra, N.J., Huang, X., Hu, S.-M.: Repfinder: finding approximately repeated scene elements for image editing. In: ACM Transactions on Graphics (TOG), vol. 29, no. 4, p. 83. ACM (2010)
Cheng, M.-M., Hou, Q.-B., Zhang, S.-H., Rosin, P.L.: Intelligent visual media processing: when graphics meets vision. J. Comput. Sci. Technol. 32(1), 110–121 (2017)
Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2376–2383 (2010)
Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)
Cheng, M., Zhang, G., Mitra, N.J., Huang, X., Hu, S.: Global contrast based salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 409–416 (2011)
Tong, N., Lu, H., Zhang, L., Ruan, X.: Saliency detection with multi-scale superpixels. IEEE Signal Process. Lett. 21(9), 1035–1039 (2014)
Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. 35, 1–17 (2019)
Wang, B., Zhang, T., Wang, X.: Salient object detection based on Laplacian similarity metrics. Vis. Comput. 34(5), 645–658 (2018)
Kapoor, A., Biswas, K., Hanmandlu, M.: An evolutionary learning based fuzzy theoretic approach for salient object detection. Vis. Comput. 33(5), 665–685 (2017)
Yang, Z., Xiong, H.: Computing object-based saliency via locality-constrained linear coding and conditional random fields. Vis. Comput. 33(11), 1403–1413 (2017)
Yang, C., Pu, J., Dong, Y., Liu, Z., Liang, L., Wang, X.: Salient object detection in complex scenes via ds evidence theory based region classification. Vis. Comput. 33(11), 1415–1428 (2017)
Li, R., Cai, J., Zhang, H., Wang, T.: Aggregating complementary boundary contrast with smoothing for salient region detection. Vis. Comput. 33(9), 1155–1167 (2017)
Zhang, Q., Lin, J., Li, W., Shi, Y., Cao, G.: Salient object detection via compactness and objectness cues. Vis. Comput. 34(4), 473–489 (2018)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Processing Systems, pp. 1106–1114 (2012)
Wang, L., Ouyang, W., Wang, X., Lu, H.: Visual tracking with fully convolutional networks. In: Proceedings of IEEE International Conference on Computer Vision, pp. 3119–3127 (2015)
Wang, L., Ouyang, W., Wang, X., Lu, H.: STCT: sequentially training convolutional networks for visual tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1373–1381 (2016)
Wang, L., Lu, H. Ruan, X., Yang, M.: Deep networks for saliency detection via local estimation and global search. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3183–3192 (2015)
Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5455–5463 (2015)
Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1265–1274 (2015)
Wang, L., Wang, L., Lu, H., Zhang, P., Ruan, X.: Saliency detection with recurrent fully convolutional networks. In: Proceedings of European Conference on Computer Vision, pp. 825–841 (2016)
Zhu, L., Hu, X., Fu, C.-W., Qin, J., Heng, P.-A.: Saliency-aware texture smoothing. In: IEEE Transactions on Visualization and Computer Graphics (2018)
Li, X., Yang, F., Cheng, H., Liu, W., Shen, D.: Contour knowledge transfer for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 355–370 (2018)
Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 234–250 (2018)
Wang, W., Shen, J., Shao, L., et al.: Correspondence driven saliency transfer. IEEE Trans. Image Process. 25(11), 5025–5034 (2016)
Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1162 (2013)
Wang, L., Lu, H., Ruan, X., Yang, M.-H.: Deep networks for saliency detection via local estimation and global search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3183–3192 (2015)
Li, G., Xie, Y., Lin, L., Yu, Y.: Instance-level salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2386–2395 (2017)
Chen, X., Zheng, A., Li, J., Lu, F.: Look, perceive and segment: finding the salient objects in images via two-stream fixation-semantic cnns. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1050–1058 (2017)
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.E., Fu, C., Berg, A.C.: SSD: single shot multibox detector. In: Proceedings of European Conference on Computer Vision, pp. 21–37 (2016)
Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. CoRR, vol. arXiv:1612.03144 (2016)
Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1741–1750 (2018)
Zhang, X., Wang, T., Qi, J., Lu, H., Wang, G.: Progressive attention guided recurrent network for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 714–722 (2018)
Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., Li, S.: Salient object detection: a discriminative regional feature integration approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2083–2090 (2013)
W, W., J, S.: Deep visual attention prediction. IEEE Trans. Image Process. 27(5), 2368–2378 (2017)
W, W., J, S., Y, Y., et al.: Stereoscopic thumbnail creation via efficient stereo saliency detection. IEEE Trans. Vis. Comput. Graph. 23(8), 2014–2027 (2016)
W, W., J, S., L, S.: Video salient object detection via fully convolutional networks. IEEE Trans. Image Process. 27(1), 38–49 (2017)
Wang, W., Shen, J., Guo, F., et al.: Revisiting video saliency: a large-scale benchmark and a new model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4894–4903 (2018)
W, W., J, S., R, Y., et al.: Saliency-aware video object segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 20–33 (2017)
Borji, A., Cheng, M.-M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE Trans. Image Process. 24(12), 5706–5722 (2015)
Wand, W., Shen, J., Dong, X., et al.: Salient object detection driven by fixation prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1711–1720 (2018)
Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)
Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.: Saliency detection via graph-based manifold ranking. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23–28, 2013, pp. 3166–3173 (2013)
Li, X., Lu, H., Zhang, L., Ruan, X., Yang, M.: Saliency detection via dense and sparse reconstruction. In: Proceedings of IEEE International Conference on Computer Vision, pp. 2976–2983 (2013)
Li, X., Zhao, L., Wei, L., Yang, M., Wu, F., Zhuang, Y., Ling, H., Wang, J.: Deepsaliency: multi-task deep neural network model for salient object detection. IEEE Trans. Image Process. 25(8), 3919–3930 (2016)
Li, G., Yu, Y.: Deep contrast learning for salient object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 478–487 (2016)
Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: R3net: recurrent residual refinement network for saliency detection. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 684–690. AAAI Press (2018)
Liu, N., Han, J.: Dhsnet: deep hierarchical saliency network for salient object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 678–686 (2016)
Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203–3212 (2017)
Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multi-level convolutional features for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 202–211 (2017)
Hu, X., Zhu, L., Qin, J., Fu, C.-W., Heng, P.-A.: Recurrently aggregating deep features for salient object detection. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Liu, J.J., Hou, Q., Cheng, M.M., et al.: A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Qin, X., Zhang, Z., Huang, C., et al.: Basnet: boundary-aware salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Hu, X., Fu, C.W., Zhu, L., et al.: Sac-net: spatial attenuation context for salient object detection. arXiv preprint arXiv:1903.10152 (2019)
Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Borji, A., Sihite, D.N., Itti, L.: Salient object detection: a benchmark. In: Computer Vision—ECCV 2012—12th European Conference on Computer Vision, Florence, Italy, October 7–13, 2012, Proceedings, Part II, pp. 414–429 (2012)
Mai, L., Niu, Y., Liu, F.: Saliency aggregation: a data-driven approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1131–1138 (2013)
Tikhonov, A.N., Arsenin, V.Y.: Solution of ill-posed problems. Math. Comput. 32(144), 491–491 (1977)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, vol. arXiv:1409.1556 (2014)
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A.C., Bengio, Y.: Maxout networks. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013, pp. 1319–1327 (2013)
González, R.C., Woods, R.E., Eddins, S.L.: Digital Image Processing Using MATLAB. Pearson-Prentice-Hall, Upper Saddle River (2004)
Li, Y., Hou, X., Koch, C., Rehg, J.M., Yuille, A.L.: The secrets of salient object segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 280–287 (2014)
Achanta, R., Hemami, S.S., Estrada, F.J., Süsstrunk, S.: Frequency-tuned salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604 (2009)
Cheng, M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R.B., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. CoRR, vol. arXiv:1408.5093 (2014)
Peng, H., Li, B., Ling, H., Hu, W., Xiong, W., Maybank, S.J.: Salient object detection via structured matrix decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 818–832 (2017)
Tong, N., Lu, H., Ruan, X., Yang, M.: Salient object detection via bootstrap learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1884–1892 (2015)
Jiang, B., Zhang, L., Lu, H., Yang, C., Yang, M.: Saliency detection via absorbing Markov chain. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1665–1672 (2013)
Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2814–2821 (2014)
Zhang, J., Sclaroff, S., Lin, Z.L., Shen, X., Price, B.L., Mech, R.: Minimum barrier salient object detection at 80 FPS. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1404–1412 (2015)
Qin, Y., Lu, H., Xu, Y., Wang, H.: Saliency detection via cellular automata. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 110–119 (2015)
Lee, G., Tai, Y., Kim, J.: Deep saliency with encoded low level distance map and high level features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 660–668 (2016)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (61871038, 61871039) and Beijing Natural Science Foundation (4182022).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liang, Y., Liu, H. & Ma, N. A novel deep network and aggregation model for saliency detection. Vis Comput 36, 1883–1895 (2020). https://doi.org/10.1007/s00371-019-01781-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-019-01781-9