Switchable CNNs for in-loop restoration and super-resolution for AV2

Urvang Joshi; Yue Chen; Innfarn Yoo; Shan Li; Feng Yang; Debargha Mukherjee

doi:10.1117/12.2681954

4 October 2023 Switchable CNNs for in-loop restoration and super-resolution for AV2

Urvang Joshi, Yue Chen, Innfarn Yoo, Shan Li, Feng Yang, Debargha Mukherjee

Proceedings Volume 12674, Applications of Digital Image Processing XLVI; 126740I (2023) https://doi.org/10.1117/12.2681954
Event: SPIE Optical Engineering + Applications, 2023, San Diego, California, United States

Abstract

In video codecs, CNN-based models have shown huge promise in two related tasks: in-loop restoration and frame super-resolution. In our previous work, we presented a framework that uses a common CNN architecture with downloadable model parameters for both these tasks with a preliminary performance study, where encoderside selection of scale factor was left as future work. The advantage of a common architecture with switchable parameters is that a single hardware inference engine can be utilized in all cases of same-resolution and super-resolution restoration, thereby limiting implementation costs. In this paper, we fully integrate this framework into the under-development AV2 video codec from the Alliance for Open Media (AOM). We also implement an algorithm for encoder-side selection of the super-resolution scale factor. With this implementation, we are able to achieve combined compression improvement up to −3.5% (AI) and −3.9% (RA) in BDRATE PSNR-Y and up to −7.8% (AI) and −7.9% (RA) in BDRATE VMAF, with inference cost as low as 1500 MACs/pixel.

Conference Presentation

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Urvang Joshi, Yue Chen, Innfarn Yoo, Shan Li, Feng Yang, and Debargha Mukherjee "Switchable CNNs for in-loop restoration and super-resolution for AV2", Proc. SPIE 12674, Applications of Digital Image Processing XLVI, 126740I (4 October 2023); https://doi.org/10.1117/12.2681954

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available