research-article

Perceptual loss guided Generative adversarial network for saliency detection

Authors:

Rung-Ching Chen,

Hui YuAuthors Info & Claims

Volume 654, Issue C

https://doi.org/10.1016/j.ins.2023.119625

Published: 27 February 2024 Publication History

Abstract

In this work, we introduce a novel approach for saliency detection through the utilization of a generative adversarial network guided by perceptual loss. Achieving effective saliency detection through deep learning entails intricate challenges influenced by a multitude of factors, with the choice of loss function playing a pivotal role. Previous studies usually formulate loss functions based on pixel-level distances between predicted and ground-truth saliency maps. However, these formulations don’t explicitly exploit the perceptual attributes of objects, such as their shapes and textures, which serve as critical indicators of saliency. To tackle this deficiency, we propose an innovative loss function that capitalizes on perceptual features derived from the saliency map. Our approach has been rigorously evaluated on six benchmark datasets, demonstrating competitive performance when compared against the forefront methods in terms of both Mean Absolute Error (MAE) and F-measure. Remarkably, our experiments reveal consistent outcomes when assessing the perceptual loss using either grayscale saliency maps or saliency-masked colour images. This observation underscores the significance of shape information in shaping the perceptual saliency cues.

The code is available at https://github.com/XiaoxuCai/PerGAN.

References

[1]

W. Gao, S. Fan, G. Li, W. Lin, A Thorough benchmark and a new model for light field saliency detection, IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).

[2]

Y. Wang, W. Zhang, L. Wang, T. Liu, and H. Lu. Multi-source uncertainty mining for deep unsupervised saliency detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11727–11736.

[3]

M.M. Cheng, N.J. Mitra, X. Huang, P.H.S. Torr, S.M. Hu, Global contrast based salient region detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (3) (2015) 569–582.

Digital Library

[4]

M. Jian, H. Yu, Towards reliable object representation via sparse directional patches and spatial center cues, Fundamental Research (2023).

[5]

Q. Liu, X. Hong, B. Zou, J. Chen, Z. Chen, G. Zhao, Hierarchical contour closure-based holistic salient object detection, IEEE Transactions on Image Processing 26 (9) (2017) 4537–4552.

[6]

P. Zhang, D. Wang, H. Lu, H. Wang, B. Yin, Learning uncertain convolutional features for accurate saliency detection, in: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 212–221.

[7]

L. Wang, L. Wang, H. Lu, P. Zhang, X. Ruan, Saliency detection with recurrent fully convolutional networks, in: European Conference on Computer Vision (ECCV), 2016, pp. 825–841.

[8]

C. Zhang, F. Yang, G. Qiu, Q. Zhang, Salient object detection with capsule-based conditional generative adversarial network, in: 2019 IEEE International Conference on Image Processing (ICIP), 2019.

[9]

M. Arjovsky, S. Chintala, L. Bottou, Wasserstein generative adversarial networks, in: Proceedings of International Conference on Machine Learning (ICML), 2017, pp. 214–223.

[10]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, X.u. Bing, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Communications of the ACM 63 (11) (2020) 139–144.

Digital Library

[11]

J. Johnson, A. Alahi, F.F. Li, Perceptual losses for real-time style transfer and super-resolution, in: European conference on computer vision (ECCV), 2016, pp. 694–711.

[12]

G. Lee, Y.W. Tai, J. Ki, ELD-net: An efficient deep learning architecture for accurate saliency detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (7) (2017) 1599–1610.

[13]

X. Zhou, W. Cao, H. Gao, Z. Ming, J. Zhang, STI-Net: Spatiotemporal integration network for video saliency detection, Information Sciences 628 (2023) 134–147.

[14]

L. Wei, G. Zong, EGA-Net: Edge feature enhancement and global information attention network for RGB-D salient object detection, Information Sciences 626 (2023) 223–248.

[15]

P. Zhang, D. Wang, H. Lu, H. Wang, X. Ruan, Amulet: Aggregating multi-level convolutional features for salient object detection, in: Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017, pp. 202–211.

[16]

L. Wang, L. Wang, H. Lu, P. Zhang, X. Ruan, Salient object detection with recurrent fully convolutional networks, IEEE Transactions on Pattern Analysis and Machine Intelligence. 41 (7) (2018) 1734–1746.

[17]

T. Zhao, X. Wu, Pyramid Feature Attention Network for Saliency Detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3085–3094.

[18]

Q. Wang, J. Lin, Y. Yuan, Salient band selection for hyperspectral image classification via manifold ranking, IEEE Transactions on Neural Networks and Learning Systems. 27 (6) (2016) 1279–1289.

[19]

Y. Liu, Q. Zhang, D. Zhang, J. Han. Employing deep part-object relationships for salient object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV), 2019, pp. 1232–1241.

[20]

S. Song, Z. Jia, J. Yang, N. Kasabov, Salient detection via the fusion of background-based and multiscale frequency-domain features, Information Sciences 618 (2022) 53–71.

[21]

D. Zhang, J. Han, Y. Zhang, D. Xu, Synthesizing supervision for learning deep saliency network without human annotation, IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (7) (2019) 1755–1769.

[22]

P. Isola, J. Zhu, T. Zhou, A.A. Efros, Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1125–1134.

[23]

H. Pan, X. Niu, R. Li, S. Shen, Y. Dou, Supervised adversarial networks for image saliency detection, International Society for Optics and Photonics 11373 (2020) 113730H.

[24]

X. Cai, and H. Yu, “Saliency detection by conditional generative adversarial network” International Society for Optics and Photonics, 10615(2018) 1061541.

[25]

J. Wu, X. Liu, Q. Lu, Z. Lin, N. Qin, Q. Shi, FW-GAN: Underwater image enhancement using generative adversarial network with multi-scale fusion, Signal Processing: Image Communication 109 (2022).

[26]

Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, F. Durand, What do different evaluation metrics tell us about saliency models, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (3) (2019) 740–757.

[27]

S. Jetley, N. Murray, E. Vig, End-to-end saliency mapping via probability distribution prediction, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5753–5761.

[28]

C. Wang, C. Xu, C. Wang, D. Tao, Perceptual adversarial networks for image-to-image transformation, IEEE Transactions on Image Processing. 27 (8) (2018) 4066–4079.

[29]

F. Zhan, Y.Yu, K. Cui, G. Zhang, S. Lu, J. Pan, C. Zhang, F. Ma, X.Xie, C. Miao, Unbalanced feature transport for exemplar-based image translation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 15028–15038.

[30]

Q. Yan, L. Xu, J. Shi, J. Jia, Hierarchical saliency detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1155–1162.

[31]

Y. Li, X. Hou, C. Koch, J.M. Rehg, A.L. Yuille, The secrets of salient object segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 280–287.

[32]

G. Li, Y. Yu, Visual saliency based on multiscale deep features, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 5455–5463.

[33]

L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, X. Ruan, Learning to detect salient objects with image-level supervision, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 136–145.

[34]

C. Yang, L. Zhang, H. Lu, X. Ruan, M.H. Yang, Saliency detection via graph-based manifold ranking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 3166–3173.

[35]

M. Cheng, N.J. Mitra, X. Huang, S.M. Hu, Salientshape: group saliency in image collections, The Visual Computer. 30 (4) (2014) 443–453.

[36]

R. Zhao, W. Ouyang, H. Li, X. Wang, Saliency detection by multi-context deep learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1265–1274.

[37]

P. Zhang, D. Wang, H. Lu, H. Wang, B. Yin, Learning uncertain convolutional features for accurate saliency detection, in: Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017, pp. 212–221.

[38]

G. Li, Y. Yu, Deep contrast learning for salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 478–487.

[39]

N. Liu, J. Han, Dhsnet: Deep hierarchical saliency network for salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 678–686.

[40]

D. Sun, H. Wu, Z. Ding, S. Li, B. Luo, Salient object detection based on deep multi-level cascade network, in: International Conference on Brain Inspired Cognitive Systems, 2019, pp. 86–95.

[41]

X.i. Li, L. Zhao, L. Wei, M.-H. Yang, F. Wu, Y. Zhuang, H. Ling, J. Wang, Deepsaliency: Multi-task deep neural network model for salient object detection, IEEE Transactions on Image Processing 25 (8) (2016) 3919–3930.

[42]

T. Wang, L. Zhang, H. Lu, C. Sun, J. Qi, Kernelized subspace ranking for saliency detection, European Conference on Computer Vision (ECCV) (2016) 450–466.

[43]

M. Qian, J. Qi, L. Zhang, M. Feng, H. Lu, Language-aware weak supervision for salient object detection, Pattern Recognition 96 (2019) 106955.

[44]

L. Wang, H. Lu, X. Ruan, M.-H. Yang, Deep networks for saliency detection via local estimation and global search, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3183–3192.

[45]

Y. Liang, H. Liu, N. Ma, A novel deep network and aggregation model for saliency detection, The Visual Computer 36 (9) (2019) 1–13.

[46]

F. Cao, Y. Liu, D. Wang, Efficient saliency detection using convolutional neural networks with feature selection, Information Sciences 456 (2018) 34–49.

[47]

N. Tong, H. Lu, X. Ruan, M.-H. Yang, Salient object detection via bootstrap learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1884–1892.

[48]

M.-U. Hassan, D. Niu, X. Zhao, M.-S. Ahamed Shohag, Y. Ma, M. Zhang, Salient object detection based on cnn fusion of two types of saliency models, in: IEEE 2019 International Conference on Image and Vision Computing New Zealand (IVCNZ), 2019, pp. 1–6.

[49]

H. Jiang, J. Wang, Z. Yuan, Y. Wu, N. Zheng, S. Li, SAlient Object Detection: A DiscriminAtive RegionAl FeAture IntegrAtion ApproAch, 2013, pp. 2083–2090.

Index Terms

Perceptual loss guided Generative adversarial network for saliency detection
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object detection
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Object-level saliency detection with color attributes

Recently, saliency detection has been becoming a popular topic in computer vision. In this paper we propose an object-level saliency detection algorithm which explicitly explores bottom-up visual attention and objectness cues. Firstly, some category-...
Saliency detection using boundary information

Efficient and robust saliency detection is a fundamental problem in computer vision field for its wide applications, such as image segmentation and image retargeting, etc. In this paper, with the aim of uniformly highlighting the salient objects and ...
Merging fixation for saliency detection in a multilayer graph

In this paper, a multilayer graph-based saliency detection algorithm by merging fixation is proposed to effectively detect salient objects in complex scenes. First, the fixation location of an image is acquired by using fixation prediction models. This ...

Comments

Information & Contributors

Information

Published In

cover image Information Sciences: an International Journal

Information Sciences: an International Journal Volume 654, Issue C

Jan 2024

1081 pages

Issue’s Table of Contents

Copyright © 2023.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 27 February 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents