research-article

CMAIR: content and mask-aware image retargeting

Authors:

Hon-Hang Chang,

Timothy K. Shih,

Wallapak TavanapongAuthors Info & Claims

Volume 78, Issue 15

Pages 21731 - 21758

https://doi.org/10.1007/s11042-019-7462-2

Published: 01 August 2019 Publication History

Abstract

In recent years, more and more image retargeting techniques have been proposed to facilitate our daily life, in particular those based on the use of seam carving, warping or the combination of them. However, these techniques can only retarget the source picture into the same shape of a square, and these approaches cannot be reshape into a circular, a polygon or other shapes. This paper focuses on creating a graphics editing system, named CMAIR (Content and Mask-Aware Image Retargeting), for image retargeting, which can retarget the source images into different shapes of image to highlight the salient objects of primary region of interest. CMAIR effectively supports removal of unimportant pixels, and frames as many surrounding objects inside the provided mask as possible. Also, we propose a unique irregular interpolation method to produce four possible target images, and an evaluation mechanism to decide the best candidate image as the final output with the consideration of image saliency. The results show that not only the source image can be placed into different targeted shapes of mask, but also the salient objects are retained and highlighted as much as possible.

References

[1]

Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. In: ACM Transactions on Graphics (TOG), vol 26. ACM, pp 10

Digital Library

[2]

Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):24–1

Digital Library

[3]

Bo Y, Sun K, Liu L (2013) Matching-area-based seam carving for video retargeting. IEEE Trans Circuits Syst Video Technol 23(2):302–310

[4]

Bo Y, Yuan B, Bo Y (2014) Effective video retargeting with jittery assessment. IEEE Trans Multimedia 16(1):272–277

[5]

Castillo S, Judd T, Gutierrez D (2011) Using eye-tracking to assess different image retargeting methods. In: Proceedings of the ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization, ACM, pp 7–14

[6]

Cho S, Choi H, Matsushita Y, Lee S (2009) Image retargeting using importance diffusion. In: 2009 16th IEEE International Conference on Image processing (ICIP), IEEE, pp 977–980

[7]

Deng C, Lin W, Cai J (2012) Content-based image compression for arbitrary-resolution display devices. IEEE Trans Multimedia 14(4):1127–1139

Digital Library

[8]

Dong W, Zhou Ning, Paul J-C, Zhang X (2009) Optimized image resizing using seam carving and scaling. In: ACM Transactions on Graphics (TOG), vol 28, ACM, pp 125

[9]

Fan X, Xie X, Zhou H-Q, Ma W-Y (2003) Looking into video frames on small displays. In: Proceedings of the Eleventh ACM International Conference on Multimedia, ACM, pp 247–250

[10]

Fang Y, Chen Z, Lin W, Lin C-W (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process 21(9):3888–3901

Digital Library

[11]

Fang Y, Lin W, Lee B-S, Lau C-T, Chen Z, Lin C-W (2012) Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Trans Multimedia 14(1):187–198

Digital Library

[12]

Fang Y, Zeng K, Wang Z, Lin W, Fang Z, Lin C-W (2014) Objective quality assessment for image retargeting based on structural similarity. cIEEE J Emerging Sel Top Circuits Syst 4(1):95–105

[13]

Grundmann Matthias, Kwatra Vivek, Han Mei, Essa Irfan (2010) Discontinuous seam-carving for video retargeting. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 569–576

[14]

Guo Y, Liu F, Shi J, Zhou Z-H, Gleicher M (2009) Image retargeting using mesh parametrization. IEEE Trans Multimedia 11(5):856–867

Digital Library

[15]

Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

Digital Library

[16]

Kiess J, Kopf S, Guthier B, Effelsberg W (2010) Seam carving with improved edge preservation. In: ISAndamp;t/SPIE electronic imaging, International Society for Optics and Photonics, pp 75420g–75420g

[17]

Kopf S, Guthier B, Lemelson H, Effelsberg W (2009) Adaptation of web pages and images for mobile applications. In: ISAndamp;t/SPIE Electronic Imaging, International Society for Optics and Photonics, pp 72560c–72560c

[18]

Lee J-H, Kim K, Park J, Park JY, Jung S-W (2016) Image recomposition using seam carving and insertion considering the rule of thirds. IEIE Transaction on Smart Processing and Computing 5(1):1–4

[19]

Li B, Duan L-Y, Wang J, Ji R, Lin C-W, Gao W (2014) Spatiotemporal grid flow for video retargeting. IEEE Trans Image Process 23(4):1615–1628

Digital Library

[20]

Liang Y, Liu Y-J, Gutierrez D (2017) Objective quality prediction of image retargeting algorithms. IEEE Trans Vis Comput Graph 23(2):1099–1110

Digital Library

[21]

Lin S-S, Yeh I-C, Lin C-H, Lee T-Y (2013) Patch-based image warping for content-aware retargeting. IEEE Trans Multimedia 15(2):359–368

Digital Library

[22]

Liu F, Gleicher M (2005) Automatic image retargeting with fisheye-view warping. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, ACM, pp 153– 162

[23]

Liu F, Gleicher M (2006) Video retargeting: automating pan and scan. In: Proceedings of the 14th ACM International Conference on Multimedia, ACM, pp 241–250

[24]

Liu Hao, Xie Xing, Ma Wei-Ying, Zhang Hong-Jiang (2003) Automatic browsing of large pictures on mobile devices. In: Proceedings of the Eleventh ACM International Conference on Multimedia, ACM, pp 148–155

[25]

Liu Y-J, Xi L, Xuan Y-M, Chen W-F, Fu X-L (2011) Image retargeting quality assessment. In: Computer Graphics Forum, vol 30. Wiley online library, pp 583–592

[26]

Ma L, Lin W, Deng C, Ngan KN (2012) Image retargeting quality assessment: a study of subjective scores and objective metrics. IEEE J Sel Top Sign Proces 6(6):626–639

[27]

Ma L, Lin W, Deng C, Ngan KN (2012) Study of subjective and objective quality assessment of retargeted images. In: 2012 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, pp 2677–2680

[28]

Ma L, Xu L, Zhang Y, Yan Y, Ngan KN (2016) No-reference retargeted image quality assessment based on pairwise rank learning. IEEE Trans Multimedia 18 (11):2228–2237

Digital Library

[29]

Ni B, Xu M, Cheng B, Wang M, Yan S, Qi T. (2013) Learning to photograph a compositional perspective. IEEE Trans Multimedia 15(5):1138–1151

Digital Library

[30]

Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: 2009 IEEE 12th International Conference on Computer Vision, IEEE, pp 151–158

[31]

Qi S, Chi Y-TJ, Peter AM, Ho J (2016) Casair: Content and shape-aware image retargeting and its applications. IEEE Trans Image Process 25(5):2222–2232

Digital Library

[32]

Qu Z, Wang J, Xu M, Lu H (2013) Context-aware video retargeting via graph model. IEEE Trans Multimedia 15(7):1677–1687

Digital Library

[33]

Rubinstein M, Gutierrez D, Sorkine O, Shamir A (2010) A comparative study of image retargeting. In: ACM Transactions on Graphics (TOG), vol 29. ACM, pp 160

[34]

Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. In: ACM Transactions on Graphics (TOG), vol 27. ACM, pp 16

[35]

Rubinstein M, Shamir A, Avidan S (2009) Multi-operator media retargeting. In: ACM Transactions on Graphics (TOG), vol 28, ACM, pp 23

[36]

Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M (2006) Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 771–780

[37]

Shamir A, Sorkine O (2009) Visual media retargeting. In: ACM SIGGRAPH ASIA 2009 Courses, ACM, pp 11

Digital Library

[38]

Simakov Denis, Caspi Yaron, Shechtman Eli, Irani Michal (2008) Summarizing visual data using bidirectional similarity. In: 2008. CVPR 2008. IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8

[39]

Suh B, Ling H, Bederson BB, Jacobs DW (2003) Automatic thumbnail cropping and its effectiveness. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, ACM, pp 95–104

[40]

Tao C, Jia J, Sun H (2007) Active window oriented dynamic video retargeting. In: Proceedings of the Workshop on Dynamical Vision, ICCV

[41]

Wang Y-S, Fu H, Sorkine O, Lee T-Y, Seidel H-P (2009) Motion-aware temporal coherence for video resizing. ACM Trans Graph (TOG) 28(5):127

[42]

Wang Y-S, Tai C-L, Sorkine O, Lee T-Y (2008) Optimized scale-and-stretch for image resizing. In: ACM Transactions on Graphics (TOG), vol 27. ACM, pp 118

[43]

Wolf L, Guttmann M, Cohen-Or D (2007) Non-homogeneous content-driven video-retargeting. In: 2007. ICCV 2007. IEEE 11th International Conference on Computer Vision, IEEE, pp 1–6

[44]

Wu L, Cao L, Min X, Wang J (2014) A hybrid image retargeting approach via combining seam carving and grid warping. J Multimed 9(4):483–492

[45]

Wu L, Gong Y, Yuan X, Zhang X, Cao L (2014) Semantic aware sport image resizing jointly using seam carving and warping. Multimed Tools Appl 70 (2):721–739

Digital Library

[46]

Yen T-C, Tsai C-M, Lin C-W (2011) Maintaining temporal coherence in video retargeting using mosaic-guided scaling. IEEE Trans Image Process 20(8):2339–2351

[47]

Zhang L, Wang M, Nie L, Hong L, Rui Y, Qi T (2015) Retargeting semantically-rich photos. IEEE Trans Multimedia 17(9):1538–1549

Digital Library

Cited By

Ma YYu ZZhou YXu CYu D(2023)Visual saliency detection via invariant feature constrained stacked denoising autoencoderMultimedia Tools and Applications10.1007/s11042-023-14525-882:18(27451-27472)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1007/s11042-023-14525-8
Tang ZYao JZhang QLuo Y(2023)Multi-operator image retargeting with visual quality preservation of salient regionsMultimedia Systems10.1007/s00530-022-01023-429:2(811-829)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1007/s00530-022-01023-4
Tang ZYao JZhang Q(2022)Multi-operator image retargeting in compressed domain by preserving aspect ratio of important contentsMultimedia Tools and Applications10.1007/s11042-021-11376-z81:1(1501-1522)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1007/s11042-021-11376-z

Recommendations

Salient edge and region aware image retargeting

The purpose of image retargeting is to automatically adapt a given image to fit the size of various displays without introducing severe visual distortions. The seam carving method can effectively achieve this task and it needs to define image importance ...
Image retargeting with a 3D saliency model

Image retargeting is an attractive topic since variety devices are with different resolutions. So far, there are two main strategies for this problem. One is removing or adding appropriate pixels in image directly, such as the seam carving approach, and ...
The Multi-modality Content-aware Retargeting Algorithm and Variable Scale Similarity Measurement for Image Retargeting
ICAIP '18: Proceedings of the 2nd International Conference on Advances in Image Processing

Image retargeting technique requires important information preservation and less edge distortion during increasing/decreasing image retargeting. In order to preserve the important contents, the saliency feature and depth feature are used to ...

Comments

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 78, Issue 15

Aug 2019

1603 pages

ISSN:1380-7501

Issue’s Table of Contents

Copyright © 2019 Springer Science+Business Media, LLC, part of Springer Nature.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 August 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ma YYu ZZhou YXu CYu D(2023)Visual saliency detection via invariant feature constrained stacked denoising autoencoderMultimedia Tools and Applications10.1007/s11042-023-14525-882:18(27451-27472)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1007/s11042-023-14525-8
Tang ZYao JZhang QLuo Y(2023)Multi-operator image retargeting with visual quality preservation of salient regionsMultimedia Systems10.1007/s00530-022-01023-429:2(811-829)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1007/s00530-022-01023-4
Tang ZYao JZhang Q(2022)Multi-operator image retargeting in compressed domain by preserving aspect ratio of important contentsMultimedia Tools and Applications10.1007/s11042-021-11376-z81:1(1501-1522)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1007/s11042-021-11376-z

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents