research-article

Image retargeting with a 3D saliency model

Authors:

Meng WangAuthors Info & Claims

Signal Processing, Volume 112, Issue C

Pages 53 - 63

https://doi.org/10.1016/j.sigpro.2014.11.001

Published: 01 July 2015 Publication History

Abstract

Image retargeting is an attractive topic since variety devices are with different resolutions. So far, there are two main strategies for this problem. One is removing or adding appropriate pixels in image directly, such as the seam carving approach, and the other is warping the mesh of image and then mapping the texture. However, existing approaches may encounter difficulties on different types of images since most of them only consider 2D features. In this paper, we propose an adaptive 3D saliency model based on the 3D structure of images to incorporate 3D information into both seam carving and mesh warping for image retargeting. Considering the characteristics of the two strategies, the 3D information are used in different ways. In seam caving, the adaptive 3D saliency is combined with L1 norm of gradient to generate an energy map for searching the least important seam. In warping, depth information is explored to improve the detection of important edges in mesh generation. Experimental results demonstrate the advantages of our method in both the seam carving and the image warping algorithms. HighlightsWe propose to explore 3D information in image retargeting.In seam caving, 3D saliency works with L-1 norm of gradient to generate energy map for seam searching.In warping, 3D information is explored to build structure constraint and 3D saliency adjusts weight of mesh edges in warping.

References

[1]

L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach. Intell., 20 (1998) 1254-1259.

Digital Library

[2]

L. Itti, C. Koch, Feature combination strategies for saliency-based visual attention systems, J. Electron. Imaging, 10 (2001) 161-169.

[3]

M.A. Hasan, C. Kim, An automatic image browsing technique for small display users, in: 11th International Conference on Advanced Communication Technology, 2009. ICACT 2009, vol. 3, IEEE, 2009, pp. 2044-2049.

[4]

Y. Liang, Z. Su, C. Wang, D. Wang, X. Luo, Optimised image retargeting using aesthetic-based cropping and scaling, IET Image Process., 7 (2013) 61-69.

[5]

V. Setlur, T. Lechner, M. Nienhaus, B. Gooch, Retargeting images and video for preserving information saliency, IEEE Comput. Graph. Appl., 27 (2007) 80-88.

Digital Library

[6]

R. Gal, O. Sorkine, D. Cohen-Or, Feature-aware texturing, in: Proceedings of the 17th Eurographics Conference on Rendering Techniques, Eurographics Association, 2006, pp. 297-303.

[7]

L. Wolf, M. Guttmann, D. Cohen-Or, Non-homogeneous content-driven video-retargeting, in: IEEE 11th International Conference on Computer Vision, 2007. ICCV 2007, IEEE, 2007, pp. 1-6.

[8]

S. Avidan, A. Shamir, Seam carving for content-aware image resizing, in: ACM Transactions on Graphics (TOG), vol. 26, ACM, 2007, p. 10.

[9]

M. Rubinstein, A. Shamir, S. Avidan, Multi-operator media retargeting, in: ACM Transactions on Graphics (TOG), vol. 28, ACM, 2009, p. 23.

Digital Library

[10]

R. Achanta, S. Susstrunk, Saliency detection for content-aware image resizing, in: 2009 16th IEEE International Conference on Image Processing (ICIP), IEEE, 2009, pp. 1005-1008.

[11]

M. Wang, R. Hong, X.-T. Yuan, S. Yan, T.-S. Chua, Movie2comics, IEEE Trans. Multimed., 14 (2012) 858-870.

Digital Library

[12]

M. Wang, R. Hong, G. Li, Z.-J. Zha, S. Yan, T.-S. Chua, Event driven web video summarization by tag localization and key-shot identification, IEEE Trans. Multimed., 14 (2012) 975-985.

Digital Library

[13]

U. Rajashekar, I. van der Linde, A.C. Bovik, L.K. Cormack, Gaffe, IEEE Trans. Image Process., 17 (2008) 564-573.

[14]

R. Achanta, S. Hemami, F. Estrada, S. Susstrunk, Frequency-tuned salient region detection, in: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE, 2009, pp. 1597-1604.

[15]

B. Schauerte, R. Stiefelhagen, Predicting human gaze using quaternion DCT image signature saliency and face detection, in: 2012 IEEE Workshop on Applications of Computer Vision (WACV), IEEE, 2012, pp. 137-144.

[16]

C. Guo, Q. Ma, L. Zhang, Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform, in: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE, 2008, pp. 1-8.

[17]

M. Cerf, J. Harel, W. Einhäuser, C. Koch, Predicting human gaze using low-level saliency combined with face detection, in: NIPS, 2007.

[18]

Y. Gao, M. Wang, Z.-J. Zha, Q. Tian, Q. Dai, N. Zhang, Less is more, IEEE Trans. Multimed., 13 (2011) 1007-1018.

Digital Library

[19]

Y. Gao, M. Wang, D. Tao, R. Ji, Q. Dai, 3-d object retrieval and recognition with hypergraph analysis, IEEE Trans. Image Process., 21 (2012) 4290-4303.

Digital Library

[20]

Y. Gao, M. Wang, R. Ji, X. Wu, Q. Dai, 3D object retrieval with Hausdorff distance learning, IEEE Trans. Industrial Electronics, 61 (2014) 2088-2098.

[21]

Y. Gao, J. Tang, R. Hong, S. Yan, Q. Dai, N. Zhang, T.-S. Chua, Camera constraint-free view-based 3-d object retrieval, IEEE Trans. Image Process., 21 (2012) 2269-2281.

Digital Library

[22]

S. Elezovikj, H. Ling, X. Chen, Foreground and scene structure preserved visual privacy protection using depth information, in: 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), IEEE, 2013, pp. 1-4.

[23]

S.-W. Jung, S.-J. Ko, Depth sensation enhancement using the just noticeable depth difference, IEEE Trans. Image Process., 21 (2012) 3624-3637.

Digital Library

[24]

M.J. Dahan, N. Chen, A. Shamir, D. Cohen-Or, Combining color and depth for enhanced image segmentation and retargeting, Vis. Comput., 28 (2012) 1181-1193.

[25]

J. Shen, D. Wang, X. Li, Depth-Aware Image Seam Carving, IEEE Trans. Cybernetics, 43 (2013) 1453-1461.

[26]

W.-Y. Lin, P.-C. Wu, B.-R. Chen, Image Retargeting Using Depth Enhanced Saliency, Proceedings of 3D Systems and Applications(3DSA) 2013, S7-1, pp. 1-4.

[27]

C.K.J. Harel, P. Perona, Graph-based visual saliency, Adv. Neural Inf. Process. Syst., 19 (2007) 545-552.

[28]

D.-S. Hwang, S.-Y. Chien, Content-aware image resizing using perceptual seam carving with human attention model, in: 2008 IEEE International Conference on Multimedia and Expo, IEEE, 2008, pp. 1029-1032.

[29]

M. Grundmann, V. Kwatra, M. Han, I. Essa, Discontinuous seam-carving for video retargeting, in: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2010, pp. 569-576.

[30]

Y. Jin, L. Liu, Q. Wu, Nonhomogeneous scaling optimization for realtime image resizing, Vis. Comput., 26 (2010) 769-778.

Digital Library

[31]

J. Wang, M. Perreira Da Silva, P. Le Callet, V. Ricordel, Computational model of stereoscopic 3D visual saliency, IEEE Trans. Image Process., 22 (2013) 2151-2162.

Digital Library

[32]

C. Lang, T.V. Nguyen, H. Katti, K. Yadati, M. Kankanhalli, S. Yan, Depth matters: influence of depth cues on visual saliency, in: Computer Vision-ECCV 2012, Springer, 2012, pp. 101-115.

[33]

D. Scharstein, C. Pal, Learning conditional random fields for stereo, in: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR¿07. IEEE, 2007, pp. 1-8.

[34]

D. Scharstein, R. Szeliski, High-accuracy stereo depth maps using structured light, in: Proceedings of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003, vol. 1, IEEE, 2003, pp. I-195.

[35]

M. Urvoy, M. Barkowsky, R. Cousseau, Y. Koudota, V. Ricorde, P. Le Callet, J. Gutiérrez, N. García, Nama3ds1-cospad1: subjective video quality assessment database on coding conditions introducing freely available high quality 3d stereoscopic sequences, in: 2012 Fourth International Workshop on Quality of Multimedia Experience (QoMEX), IEEE, 2012, pp. 109-114.

Cited By

Ishikawa NZhu ZKim JChung WGo KMao X(2023)Seamless Image Editing for Perceptual Size Restoration Based on Seam CarvingAdvances in Computer Graphics10.1007/978-3-031-50069-5_27(322-334)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-50069-5_27
Piao YLi XZhang MYu JLu H(2019)Saliency Detection via Depth-Induced Cellular Automata on Light FieldIEEE Transactions on Image Processing10.1109/TIP.2019.294243429(1879-1889)Online publication date: 26-Nov-2019
https://dl.acm.org/doi/10.1109/TIP.2019.2942434
Wang AWang MPan GYuan X(2017)Salient object detection with high‐level prior based on Bayesian fusionIET Computer Vision10.1049/iet-cvi.2016.026311:3(199-206)Online publication date: 28-Feb-2017
https://dl.acm.org/doi/10.1049/iet-cvi.2016.0263
Show More Cited By

Image retargeting with a 3D saliency model
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Dynamic distortion maps for image retargeting

We present an image retargeting method that incorporates image content distortion into a mesh optimization process through the generation of dynamic distortion maps. The warping process is driven by the distortion produced by the warping process itself. ...
Salient edge and region aware image retargeting

The purpose of image retargeting is to automatically adapt a given image to fit the size of various displays without introducing severe visual distortions. The seam carving method can effectively achieve this task and it needs to define image importance ...
Image retargeting using depth assisted saliency map

Retargeting algorithms are used to transfer and display images on devices with various sizes and resolutions. All of these algorithms try to preserve the important parts of image against distortions while producing a retargeted image with visual quality ...

Comments

Information & Contributors

Information

Published In

cover image Signal Processing

Signal Processing Volume 112, Issue C

July 2015

209 pages

ISSN:0165-1684

Issue’s Table of Contents

Copyright © Elsevier B.V.

Publisher

Elsevier North-Holland, Inc.

United States

Publication History

Published: 01 July 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ishikawa NZhu ZKim JChung WGo KMao X(2023)Seamless Image Editing for Perceptual Size Restoration Based on Seam CarvingAdvances in Computer Graphics10.1007/978-3-031-50069-5_27(322-334)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-50069-5_27
Piao YLi XZhang MYu JLu H(2019)Saliency Detection via Depth-Induced Cellular Automata on Light FieldIEEE Transactions on Image Processing10.1109/TIP.2019.294243429(1879-1889)Online publication date: 26-Nov-2019
https://dl.acm.org/doi/10.1109/TIP.2019.2942434
Wang AWang MPan GYuan X(2017)Salient object detection with high‐level prior based on Bayesian fusionIET Computer Vision10.1049/iet-cvi.2016.026311:3(199-206)Online publication date: 28-Feb-2017
https://dl.acm.org/doi/10.1049/iet-cvi.2016.0263
Wang AWang MLi XMi ZZhou H(2017)A Two-Stage Bayesian Integration Framework for Salient Object Detection on Light FieldNeural Processing Letters10.1007/s11063-017-9610-x46:3(1083-1094)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1007/s11063-017-9610-x
Terzić KHansard M(2016)Methods for reducing visual discomfort in stereoscopic 3DImage Communication10.1016/j.image.2016.08.00247:C(402-416)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.image.2016.08.002

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents