Bringing Old Films Back to Life

We present a learning-based framework, recurrent transformer network (RTN), to restore heavily degraded old films. Instead of performing frame-wise restoration, our method is based on the hidden knowledge learned from adjacent frames that contain abundant information about the occlusion, which is beneficial to restore challenging artifacts of each frame while ensuring temporal coherency. Moreover, contrasting the representation of the current frame and the hidden knowledge makes it possible to infer the scratch position in an unsupervised manner, and such defect localization generalizes well to real-world degradations. To better resolve mixed degradation and compensate for the flow estimation error during frame alignment, we propose to leverage more expressive transformer blocks for spatial restoration. Experiments on both synthetic dataset and real-world old films demonstrate the significant superiority of the proposed RTN over existing solutions. In addition, the same framework can effectively propagate the color from keyframes to the whole video, ultimately yielding compelling restored films. The implementation and model will be released at https.//github.comiraywzy/Bringtng-Old-Films Back-to-Life.

Automated summary

We propose a learning-based framework called recurrent transformer network (RTN) for restoring heavily degraded old films. Our method utilizes the hidden knowledge learned from adjacent frames to ensure temporal coherency and effectively restore challenging artifacts. The framework also allows for unsupervised scratch position inference, which generalizes well to real-world degradations. Experimental results demonstrate the significant superiority of RTN over existing solutions on both synthetic and real-world old films.