research-article

Faster Inter Prediction by NR-Frame in VVC

Authors:

Sio-Kei ImAuthors Info & Claims

ICGSP '23: Proceedings of the 2023 7th International Conference on Graphics and Signal Processing

Pages 24 - 28

https://doi.org/10.1145/3606283.3606287

Published: 11 August 2023 Publication History

Abstract

VVC is the next generation video coding standard in which inter prediction plays an important role to reduce the redundancy between adjacent frames. The coding time is longer since larger blocks and more motion search are supported, and the accuracy of inter prediction is limited because only temporal information is used in the conventional algorithm. This work make use of YOLOv5 to refine inter prediction in VVC, introducing an architecture that combines detected objects and tracking results with the proposed NR-Frame, which perform faster prediction of coded blocks within such detected objects. The experimental results demonstrate that the proposed method can achieve an average 11.45% (up to 13.27%) reduction in coding time under RA conditions compared to VTM-13.0.

References

[1]

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4: Optimal Speed and Accuracy of Object Detection. https://doi.org/10.48550/ARXIV.2004.10934

[2]

Frank Bossen, Jill Boyce, Karsten Suehring, Xiang Li, and Vadim Seregin. 2020. VTM common test conditions and software reference configurations for SDR video. Preview document JVET-T2010 for Teleconference meeting (Nov. 2020). https://jvet-experts.org/doc_end_user/current_document.php?id=10545

[3]

Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens-Rainer Ohm. 2021. Overview of the Versatile Video Coding (VVC) Standard and its Applications. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (oct 2021), 3736–3764. https://doi.org/10.1109/tcsvt.2021.3101953

[4]

Ka-Hou Chan and Sio-Kei Im. 2021. Rounding of improved DCT transform coding for H.266/VVC. In Thirteenth International Conference on Digital Image Processing (ICDIP 2021), Xudong Jiang and Hiroshi Fujita (Eds.). SPIE. https://doi.org/10.1117/12.2601046

[5]

Jinyoung Choi and Bohyung Han. 2020. Task-Aware Quantization Network for JPEG Image Compression. In Computer Vision – ECCV 2020. Springer International Publishing, 309–324. https://doi.org/10.1007/978-3-030-58565-5_19

Digital Library

[6]

Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2015. Compression Artifacts Reduction by a Deep Convolutional Network. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2015.73

Digital Library

[7]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. In Computer Vision – ECCV 2014. Springer International Publishing, 184–199. https://doi.org/10.1007/978-3-319-10593-2_13

[8]

Lingyu Duan, Jiaying Liu, Wenhan Yang, Tiejun Huang, and Wen Gao. 2020. Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics. IEEE Transactions on Image Processing 29 (2020), 8680–8695. https://doi.org/10.1109/tip.2020.3016485

Digital Library

[9]

Samira Hayat, Evşen Yanmaz, Christian Bettstetter, and Timothy X. Brown. 2020. Multi-objective drone path planning for search and rescue with quality-of-service requirements. Autonomous Robots 44, 7 (jul 2020), 1183–1198. https://doi.org/10.1007/s10514-020-09926-9

Digital Library

[10]

Jingbo He, Xiaohai He, Mozhi Zhang, Shuhua Xiong, and Honggang Chen. 2022. Deep dual-domain semi-blind network for compressed image quality enhancement. Knowledge-Based Systems 238 (feb 2022), 107870. https://doi.org/10.1016/j.knosys.2021.107870

Digital Library

[11]

Yu-Wen Huang, Jicheng An, Han Huang, Xiang Li, Shih-Ta Hsiang, Kai Zhang, Han Gao, Jackie Ma, and Olena Chubach. 2021. Block Partitioning Structure in the VVC Standard. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (oct 2021), 3818–3833. https://doi.org/10.1109/tcsvt.2021.3088134

[12]

Sio-Kei Im and Ka-Hou Chan. 2015. Multi-lambda search for improved rate-distortion optimization of H.265/HEVC. In 2015 10th International Conference on Information, Communications and Signal Processing (ICICS). IEEE. https://doi.org/10.1109/icics.2015.7459952

[13]

Sio-Kei Im and Ka-Hou Chan. 2017. Efficient mode decision with enhanced sampling algorithm for HEVC. In 2017 IEEE 18th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE. https://doi.org/10.1109/wowmom.2017.7974328

[14]

Sio-Kei Im and Ka-Hou Chan. 2022. A propagation model for package loss refinement in VVC. Electronics Letters 58, 20 (aug 2022), 759–761. https://doi.org/10.1049/ell2.12586

[15]

Wei Ke and Ka-Hou Chan. 2022. Improving Quantization Matrices for Image Coding by Machine Learning. In Proceedings of the 6th International Conference on Digital Signal Processing. ACM. https://doi.org/10.1145/3529570.3529590

Digital Library

[16]

M. I. Khalil. 2010. Image Compression Using New Entropy Coder. International Journal of Computer Theory and Engineering (2010), 39–41. https://doi.org/10.7763/ijcte.2010.v2.114

[17]

Dong Liu, Yue Li, Jianping Lin, Houqiang Li, and Feng Wu. 2020. Deep Learning-Based Video Coding. Comput. Surveys 53, 1 (feb 2020), 1–35. https://doi.org/10.1145/3368405

Digital Library

[18]

Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. 2022. Transformer-based Image Compression. In 2022 Data Compression Conference (DCC). IEEE. https://doi.org/10.1109/dcc52660.2022.00080

[19]

Simon Niklaus and Feng Liu. 2020. Softmax Splatting for Video Frame Interpolation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr42600.2020.00548

[20]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

Digital Library

[21]

Jonathan Pfaff, Alexey Filippov, Shan Liu, Xin Zhao, Jianle Chen, Santiago De-Luxan-Hernandez, Thomas Wiegand, Vasily Rufitskiy, Adarsh Krishnan Ramasubramonian, and Geert Van der Auwera. 2021. Intra Prediction and Mode Coding in VVC. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (oct 2021), 3834–3847. https://doi.org/10.1109/tcsvt.2021.3072430

[22]

Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. https://doi.org/10.48550/ARXIV.1804.02767

[23]

Wei Wang, Siyuan Hao, Yunchao Wei, Shengtao Xiao, Jiashi Feng, and Nicu Sebe. 2019. Temporal Spiking Recurrent Neural Network for Action Recognition. IEEE Access 7 (2019), 117165–117175. https://doi.org/10.1109/access.2019.2936604

[24]

Chao-Yuan Wu, Nayan Singhal, and Philipp Krähenbühl. 2018. Video Compression Through Image Interpolation. In Computer Vision – ECCV 2018. Springer International Publishing, 425–440. https://doi.org/10.1007/978-3-030-01237-3_26

Digital Library

[25]

Kai Zhang, Yi-Wen Chen, Li Zhang, Wei-Jung Chien, and Marta Karczewicz. 2019. An Improved Framework of Affine Motion Compensation in Video Coding. IEEE Transactions on Image Processing 28, 3 (mar 2019), 1456–1469. https://doi.org/10.1109/tip.2018.2877355

Digital Library

Cited By

Li NWang ZZhang Q(2024)Fast Coding Unit Partitioning Algorithm for Video Coding Standard Based on Block Segmentation and Block Connection Structure and CNNElectronics10.3390/electronics1309176713:9(1767)Online publication date: 2-May-2024
https://doi.org/10.3390/electronics13091767
Im SChan K(2024)Dynamic estimator selection for double‐bit‐range estimation in VVC CABAC entropy codingIET Image Processing10.1049/ipr2.1298018:3(722-730)Online publication date: 2-Jan-2024
https://doi.org/10.1049/ipr2.12980

Index Terms

Faster Inter Prediction by NR-Frame in VVC
1. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Video search
2. Mathematics of computing
  1. Information theory
    1. Coding theory

Recommendations

Open-Source Toolkit for Live End-to-End 4K VVC Intra Coding
MMSys '23: Proceedings of the 14th ACM Multimedia Systems Conference

Versatile Video Coding (VVC/H.266) takes video coding to the next level by doubling the coding efficiency over its predecessors for the same subjective quality, but at the cost of immense coding complexity. Therefore, VVC calls for aggressively optimized ...
Deep Inter Prediction with Error-Corrected Auto-Regressive Network for Video Coding
Modern codecs remove temporal redundancy of a video via inter prediction, i.e., searching previously coded frames for similar blocks and storing motion vectors to save bit-rates. However, existing codecs adopt block-level motion estimation, where a block ...
Advanced template matching prediction using a motion boundary
ICIGP '19: Proceedings of the 2nd International Conference on Image and Graphics Processing

In this paper, an advanced template matching prediction is proposed to improve the coding efficiency gain. Conventional template matching prediction in joint exploration model (JEM) finds a motion vector for the current coding block by using adjacent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICGSP '23: Proceedings of the 2023 7th International Conference on Graphics and Signal Processing

June 2023

83 pages

ISBN:9798400700460

DOI:10.1145/3606283

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Macao Polytechnic University

Conference

ICGSP 2023

ICGSP 2023: 2023 The 7th International Conference on Graphics and Signal Processing

June 23 - 25, 2023

Fujisawa, Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
47
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li NWang ZZhang Q(2024)Fast Coding Unit Partitioning Algorithm for Video Coding Standard Based on Block Segmentation and Block Connection Structure and CNNElectronics10.3390/electronics1309176713:9(1767)Online publication date: 2-May-2024
https://doi.org/10.3390/electronics13091767
Im SChan K(2024)Dynamic estimator selection for double‐bit‐range estimation in VVC CABAC entropy codingIET Image Processing10.1049/ipr2.1298018:3(722-730)Online publication date: 2-Jan-2024
https://doi.org/10.1049/ipr2.12980

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents