Texture-based Error Analysis for Image Super-Resolution

Evaluation practices for image super-resolution (SR) use a single-value metric, the PSNR or SSIM, to determine model performance. This provides little insight into the source of errors and model behavior. Therefore, it is beneficial to move beyond the conventional approach and reconceptualize evaluation with interpretability as our main priority. We focus on a thorough error analysis from a variety of perspectives. Our key contribution is to leverage a texture classifier, which enables us to assign patches with semantic labels, to identify the source of SR errors both globally and locally. We then use this to determine (a) the semantic alignment of SR datasets, (b) how SR models perform on each label, (c) to what extent high-resolution (HR) and SR patches semantically correspond, and more. Through these different angles, we are able to highlight potential pitfalls and blindspots. Our overall investigation highlights numerous unexpected insights. We hope this work serves as an initial step for debugging blackbox SR networks.

Automated summary

Evaluation practices for image super-resolution typically rely on single-value metrics like PSNR or SSIM, which offer limited insight into error sources and model behavior. This study proposes a new approach focusing on interpretability, utilizing a texture classifier for in-depth error analysis to identify the sources of SR errors. By examining datasets from various perspectives, the research uncovers unexpected insights that may help in debugging blackbox SR networks.