research-article

Feedforward and Feedback Modulations Based Foveated JND Estimation for Images

Authors:

Guangtao ZhaiAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 19, Issue 5

Article No.: 154, Pages 1 - 23

https://doi.org/10.1145/3579094

Published: 16 March 2023 Publication History

Abstract

The just noticeable difference (JND) reveals the key characteristic of visual perception, which has been widely used in many perception-based image and video applications. Nevertheless, the modulatory mechanism of the human visual system (HVS) has not been fully exploited in JND threshold estimation, which results in the existing JND models not being accurate enough. In this article, by analyzing the feedforward and feedback modulatory behaviors in the HVS, an enhanced foveated JND (FJND) estimation model is proposed considering modulatory effects and masking effects in visual perception. The contributions of this article are mainly twofold. On the one hand, by analyzing the modulatory behaviors in the HVS, the modulatory mechanism is incorporated into JND estimation and a hierarchical modulation-based JND estimation framework is proposed for the first time. On the other hand, according to the response characteristics of visual neurons, modulatory effects on visual sensitivity are formulated as several modulatory factors to modulate the estimated JND threshold properly. Compared with existing models, the proposed model is developed in view of not only the masking effects but also the modulatory effects, which makes our model more consistent with the HVS. For different complex input images, experimental results show that the proposed FJND model tolerates more distortion at the same perceptual quality in comparison with other existing models.

References

[1]

USC-SIPI. 2011. The USC-SIPI Image Database. Retrieved January 5, 2023 from https://sipi.usc.edu/database/database.php/.

[2]

IRCCyN/IVC. 2013. Subjective Quality Assessment—IVC Database. Retrieved January 5, 2023 from IRCCyN/IVC Image Quality Database - QUALINET Databases.

[3]

A. J. Ahumada and H. A. Peterson. 1992. Luminance-model-based DCT quantization for color image compression. Proceedings of SPIE: Human Vision, Visual Processing, and Digital Display III 1666 (1992), 365–374.

[4]

S. Bae and K. Munchurl. 2017. A DCT-based total JND profile for spatiotemporal and foveated masking effects. IEEE Transactions on Circuits and Systems for Video Technology 27, 6 (2017), 1196–1207.

Digital Library

[5]

H. B. Barlow. 1961. Possible principle underlying the transformation of sensory messages. In Sensory Communication, W. A. Rosenblith (Ed.). MIT Press, Cambridge, MA, 217–234.

[6]

Z. Chen and C. Guillemot. 2010. Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (2010), 806–819.

Digital Library

[7]

Daniel Huber, Leopoldo Petreanu, Nima Ghitani, Sachin Ranade, Tomas Hromadka, Zach Mainen, and Karel Svoboda. 2008. Sparse optical microstimulation in barrel cortex drives learned behaviour in freely moving mice. Nature 451, 7174 (2008), 61.

[8]

H. Feldman and K. L. Friston. 2010. Attention, uncertainty, and free-energy. Frontiers in Human Neuroscience 4, 1 (2010), 125.

[9]

H. Feng, M. W. Marcellin, and A. Bilgin.2015. A methodology for visually lossless JPEG2000 compression of monochrome stereo images. IEEE Transactions on Image Processing 24, 2 (2015), 560–572.

[10]

Karl Friston. 2010. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience 11, 2 (2010), 127.

[11]

Wilson S. Geisler and Jeffrey S. Perry. 1998. Real-time foveated multiresolution system for low-bandwidth video communication. Proceedings of SPIE: Human Vision and Electronic Imaging III 3299 (1998), 294–305.

[12]

K. Gu, G. Zhai, X. Yang, and W. Zhang. 2014. Using free energy principle for blind image quality assessment. IEEE Transactions on Multimedia 17, 1 (2014), 50–63.

[13]

S. Hochstein and M. Ahissar. 2002. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron 36, 5 (2002), 791–804.

[14]

Laurent Itti and Pierre Baldi. 2009. Bayesian surprise attracts human attention. Vision Research 49, 10 (2009), 1295–1306.

[15]

ITU. 2002. Methodology for the Subjective Assessment of the Quality of Television Pictures. Technical Report ITU-R BT.500-11. ITU, Geneva, Switzerland.

[16]

T. Judd, K. Ehinger, F. Durand, and A. Torralba. 2010. Learning to predict where humans look. In Proceedings of the International Conference on Computer Vision.

[17]

Karl Friston. 2009. The free-energy principle: A rough guide to the brain? Trends in Cognitive Sciences 13, 7 (2009), 293–301.

[18]

Sehwan Ki, Sung Ho Bae, Munchurl Kim, and Hyunsuk Ko. 2018. Learning-based just-noticeable-quantization–distortion modeling for perceptual video coding. IEEE Transactions on Image Processing 27, 7 (2018), 3178–3193.

[19]

David C. Knill and Alexandre Pouget. 2004. The Bayesian brain: The role of uncertainty in neural coding and computation. Trends in Neurosciences 27, 12 (2004), 712–719.

[20]

Junlin Li, Li Yu, and Hongkui Wang. 2022. Perceptual redundancy model for compression of screen content videos. IET Image Processing 16, 6 (2022), 1724–1741.

[21]

Anmin Liu, Weisi Lin, Manoranjan Paul, Chenwei Deng, and Fan Zhang. 2010. Just noticeable difference for images with decomposition model separating edge and textured regions. IEEE Transactions on Circuits and Systems for Video Technology 20, 11 (2010), 1648–1652.

Digital Library

[22]

Huanhua Liu, Yun Zhang, Huan Zhang, Chunling Fan, and Xiaoping Fan. 2019. Deep learning-based picture-wise just noticeable distortion prediction model for image compression. IEEE Transactions on Image Processing 29 (2019), 641–656.

Digital Library

[23]

S. L. Macknik and M. S. Livingstone. 1998. Neuronal correlates of visibility and invisibility in the primate visual system. Nature Neuroscience 1, 2 (1998), 144–149.

[24]

D. J. Mannion, D. J. Kersten, C. A. Olman, and J. Foxe. 2015. Scene coherence can affect the local response to natural images in human V1. European Journal of Neuroscience 42, 11 (2015), 2895–2903.

[25]

T. Oosuga, M. Tanaka, H. Inoue, and Y. Niiyama. 2012. A study on eye fixation time distribution with and without subjective evaluation of food and related pictures. In Proceedings of the 2012 SICE Annual Conference (SICE’12).

[26]

N. Ponomarenko, O. Ieremeiev, V. Lukin, K. Egiazarian, L. Jin, J. Astola, B. Vozel, K. Chehdi, M. Carli, and F. Battisti. 2013. Color image database TID2013: Peculiarities and preliminary results. In Proceedings of the European Workshop on Visual Information Processing.

[27]

R. P. Rao and D. H. Ballard. 1999. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience 2, 11 (1999), 79–87.

[28]

Kais Rouis, Mohamed-Chaker Larabi, and Jamel Belhadj Tahar. 2018. Perceptually adaptive Lagrangian multiplier for HEVC guided rate-distortion optimization. IEEE Access 6 (2018), 33589–33603.

[29]

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2008. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision 77, 1-3 (2008), 157–173.

Digital Library

[30]

Masaki Sano. 2015. Experimental demonstration of information-to-energy conversion in small fluctuating systems. In Proceedings of the APS Meeting.

[31]

Tao Tian, Hanli Wang, Lingxuan Zuo, C.-C. Jay Kuo, and Sam Kwong. 2020. Just noticeable difference level prediction for perceptual image compression. IEEE Transactions on Broadcasting 66, 3 (2020), 690–700.

[32]

P. Wallisch and J. A. Movshon. 2008. Structure and function come unglued in the visual cortex. Neuron 8, 2 (2008), 195–197.

[33]

Hongkui Wang, Shengwei Wang, Tiansong Li, Haibing Yin, and Li Yu. 2019. Surprise based JND estimation for images. IEEE Signal Processing Letters 27 (2019), 181–184.

[34]

Hongkui Wang, Li Yu, Junhui Liang, Haibing Yin, Tiansong Li, and Shengwei Wang. 2021. Hierarchical predictive coding-based JND estimation for image compression. IEEE Transactions on Image Processing 30 (2021), 487–500.

Digital Library

[35]

Hongkui Wang, Li Yu, Shengwei Wang, Tiansong Li, and Haibing Yin. 2018. A novel foveated-JND profile based on an adaptive foveated weighting model. In Proceedings of the 2018 IEEE Conference on Visual Communications and Image Processing (VCIP’18).

[36]

Hongkui Wang, Li Yu, Haibing Yin, Tiansong Li, and Shengwei Wang. 2020. An improved DCT-based JND estimation model considering multiple masking effects. Journal of Visual Communication and Image Representation 71 (2020), 102850.

[37]

Rubin Wang and Zhikang Zhang. 2007. Energy coding in biological neural networks. Cognitive Neurodynamics 1, 3 (2007), 203–212.

[38]

Rubin Wang, Zhikang Zhang, and Guanrong Chen. 2009. Energy coding and energy functions for local activities of the brain. Neurocomputing 73, 1-3 (2009), 139–150.

Digital Library

[39]

S. Wang, L. Ma, Y. M. Fang, W. Lin, S. Ma, and W. Gao. 2016. Just noticeable difference estimation for screen content images. IEEE Transactions on Image Processing 25, 8 (2016), 3838–3851.

[40]

Zhou Wang, Ligang Lu, and A. C. Bovik. 2003. Foveation scalable video coding with automatic fixation selection. IEEE Transactions on Image Processing 12, 2 (2003), 243–254.

Digital Library

[41]

Zhenyu Wei and King N. Ngan. 2009. Spatio-temporal just noticeable distortion profile for grey scale image/video in DCT domain. IEEE Transactions on Circuits and Systems for Video Technology 19, 3 (2009), 337–346.

Digital Library

[42]

J. Wu, L. Li, W. Dong, G. Shi, W. Lin, and C.-C. Jay Kuo. 2017. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing 26, 6 (2017), 2682–2693.

Digital Library

[43]

J. Wu, W. Lin, G. Shi, X. Wang, and F. Qi. 2013. Just difference estimation for images with free energy principle. IEEE Transactions on Image Processing 15, 7 (2013), 1705–1710.

Digital Library

[44]

J. Wu, G. Shi, W. Lin, A. Liu, and F. Li. 2013. Pattern masking estimation in image with structural uncertainty. IEEE Transactions on Image Processing 22, 12 (2013), 4892–4904.

Digital Library

[45]

Kaifu Yang, Chaoyi Li, and Yongjie Li. 2014. Multi feature-based surround inhibition improves contour detection in natural images. IEEE Transactions on Image Processing 23, 12 (2014), 5020–5032.

[46]

X. Yang, W. Lin, Z. Lu, E. Ong, and S. Yao. 2005. Just noticeable distortion model and its applications in video coding. Signal Processing: Image Communication 20 (2005), 662–680.

[47]

Zhipeng Zeng, Huanqiang Zeng, Jing Chen, Jianqing Zhu, Yun Zhang, and Kai Kuang Ma. 2019. Visual attention guided pixel-wise just noticeable difference model. IEEE Access 7 (2019), 132111–132119.

[48]

G. Zhai, X. Wu, X. Yang, W. Lin, and W. Zhang. 2012. A psychovisual quality metric in free-energy principle. IEEE Transactions on Image Processing 21, 1 (2012), 41–52.

[49]

Xinfeng Zhang, Shiqi Wang, Ke Gu, Weisi Lin, Siwei Ma, and Wen Gao. 2017. Just-noticeable difference-based perceptual optimization for JPEG compression. IEEE Signal Processing Letters 24, 1 (2017), 96–100.

[50]

Yun Zhang, Huanhua Liu, You Yang, Xiaoping Fan, Sam Kwong, and C. C. Jay Kuo. 2022. Deep learning based just noticeable difference and perceptual quality prediction models for compressed video. IEEE Transactions on Circuits and Systems for Video Technology 32, 3 (2022), 1197–1212.

[51]

Mingliang Zhou, Xuekai Wei, Shiqi Wang, Sam Kwong, Chi Keung Fong, Peter Wong, Wilson Yuen, and Wei Gao. 2019. SSIM-based global optimization for CTU-level rate control in HEVC. IEEE Transactions on Multimedia 21, 8 (2019), 1921–1933.

Digital Library

Cited By

Zhang LZhou XLi DYang Z(2024)HCCNet: Hybrid Coupled Cooperative Network for Robust Indoor LocalizationACM Transactions on Sensor Networks10.1145/366564520:4(1-22)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3665645
Wang GWang HLi HYu LYin HXu HYe ZSong J(2024)A survey on just noticeable distortion estimation and its applications in video codingJournal of Visual Communication and Image Representation10.1016/j.jvcir.2023.10403498:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2023.104034
Wang HZhang LYu LYang HYin HDing SXu HWang H(2023)Learning-based JNCD prediction for quality-wise perceptual quantization in HEVCJournal of Visual Communication and Image Representation10.1016/j.jvcir.2023.10387795(103877)Online publication date: Sep-2023
https://doi.org/10.1016/j.jvcir.2023.103877
Show More Cited By

Index Terms

Feedforward and Feedback Modulations Based Foveated JND Estimation for Images
1. Computing methodologies
  1. Computer graphics
    1. Image compression

Recommendations

Perception-oriented video coding based on foveated JND model
PCS'09: Proceedings of the 27th conference on Picture Coding Symposium

The ultimate goal of video compression is to provide best visual quality of the video source by using minimum bit budget. To achieve this target, spatial, temporal and perceptual redundancies should be exploited and removed such that the video source ...
Towards Attention–aware Foveated Rendering

Foveated graphics is a promising approach to solving the bandwidth challenges of immersive virtual and augmented reality displays by exploiting the falloff in spatial acuity in the periphery of the visual field. However, the perceptual models used in ...
Method and experiments of subliminal cueing for real-world images

Unconscious attention shift triggered by a subliminal cue has been shown to be automatic; however, whether it can be brought into effect for images of real-world scenes remains to be investigated. We present a subliminal cueing method that flashes ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 19, Issue 5

September 2023

262 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3585398

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 March 2023

Online AM: 04 January 2023

Accepted: 18 December 2022

Revised: 25 October 2022

Received: 08 June 2022

Published in TOMM Volume 19, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

“Pioneer” and “Leading Goose” R&D Program of Zhejiang Province
NSFC
Natural Science Foundation of Hubei Province of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
282
Total Downloads

Downloads (Last 12 months)103
Downloads (Last 6 weeks)4

Reflects downloads up to 14 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang LZhou XLi DYang Z(2024)HCCNet: Hybrid Coupled Cooperative Network for Robust Indoor LocalizationACM Transactions on Sensor Networks10.1145/366564520:4(1-22)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3665645
Wang GWang HLi HYu LYin HXu HYe ZSong J(2024)A survey on just noticeable distortion estimation and its applications in video codingJournal of Visual Communication and Image Representation10.1016/j.jvcir.2023.10403498:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2023.104034
Wang HZhang LYu LYang HYin HDing SXu HWang H(2023)Learning-based JNCD prediction for quality-wise perceptual quantization in HEVCJournal of Visual Communication and Image Representation10.1016/j.jvcir.2023.10387795(103877)Online publication date: Sep-2023
https://doi.org/10.1016/j.jvcir.2023.103877
Mao YWu JWang XLi LDong W(2023)Transfer learning for just noticeable difference estimationInformation Sciences: an International Journal10.1016/j.ins.2023.119575648:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.ins.2023.119575
Wang HYu LYang HXu HYin HZhai GLi TKuang Z(2023)Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecsSignal Processing: Image Communication10.1016/j.image.2023.117019118(117019)Online publication date: Oct-2023
https://doi.org/10.1016/j.image.2023.117019

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents