Computer Science > Computer Vision and Pattern Recognition
[Submitted on 1 Dec 2020 (v1), revised 25 Apr 2021 (this version, v2), latest version 15 Jan 2024 (v5)]
Title:Decomposition, Compression, and Synthesis Based Video Coding: A Neural Approach Through Reference-Based Super Resolution
View PDFAbstract:In pursuit of higher compression efficiency, a potential solution is the Down-Sampling based Video Coding (DSVC) where a input video is first downscaled for encoding at a relatively lower resolution, and then decoded frames are super-resolved through deep neural networks (DNNs). However, the coding gains are often bounded due to either uniform resolution sampling induced severe loss of high-frequency component, or insufficient information aggregation across non-uniformly sampled frames in existing DSVC methods. To address this, we propose to first decompose the input video into respective spatial texture frames (STFs) at its native spatial resolution that preserve the rich spatial details, and the other temporal motion frames (TMFs) at a lower spatial resolution that retain the motion smoothness; then compress them together using any popular video coder; and finally synthesize decoded STFs and TMFs for high-fidelity video reconstruction at the same resolution as its native input. This work simply applies the bicubic sampling in decomposition and Versatile Video Coding (VVC) compliant codec in compression, and puts the focus on the synthesis part. Such cross-resolution synthesis can be facilitated by Reference-based Super-Resolution (RefSR). Specifically, a motion compensation network (MCN) is devised on TMFs to efficiently align and aggregate temporal motion features that will be jointly processed with corresponding STFs using a texture transfer network (TTN) to better augment spatial details, by which the compression and resolution re-sampling noises can be effectively alleviated with better rate-distortion (R-D) efficiency, etc.
Submission history
From: Ming Lu [view email][v1] Tue, 1 Dec 2020 17:23:53 UTC (1,826 KB)
[v2] Sun, 25 Apr 2021 06:16:32 UTC (5,300 KB)
[v3] Wed, 30 Jun 2021 05:20:27 UTC (2,808 KB)
[v4] Sat, 18 Dec 2021 05:12:12 UTC (12,499 KB)
[v5] Mon, 15 Jan 2024 13:17:58 UTC (1,633 KB)
Current browse context:
cs.CV
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.