Jointly optimizing traditional image codec and neural super-resolution

Abstract

Recently, a neural super-resolution (SR) has been on the rise due to its tremendous performance enhancement. Image compression must be accompanied before processing the neural super-resolution to reduce resource consumption for storage and network bandwidth. while it is promising to balance resource (data size and computation) and image quality by jointly considering both perspectives the image compression and the neural super-resolution, existing works optimize the compression included SR pipeline at only one side. Consequently, we present a novel framework that jointly optimizes the traditional image codec and the neural super-resolution by allocating an appropriate Quantization Parameter (QP) and selecting a proper SR model complexity according to the image region. Based on our observation that allocating Quantization Parameter has a dominant position in the relationship between choosing SR model complexity, we design our framework that operates sequentially in the order of SR-aware QP allocation and QP-aware SR model selection. Experiments demonstrate that compared to a baseline that does not any additional optimization, our approach can achieve the same image quality with saving 20% of the storage resources or 50-60% of the computing resources.