GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution
CVPR 2021, Oral
Paper
Abstract
We show that pre-trained Generative Adversarial Networks (GANs), e.g., StyleGAN, can be used as a latent bank to improve the restoration quality of large-factor image super-resolution (SR). While most existing SR approaches attempt to generate realistic textures through learning with adversarial loss, our method, Generative LatEnt bANk (GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained GAN. But unlike prevalent GAN inversion methods that require expensive image-specific optimization at runtime, our approach only needs a single forward pass to generate the upscaled image. GLEAN can be easily incorporated in a simple encoder-bank-decoder architecture with multi-resolution skip connections. Switching the bank allows the method to deal with images from diverse categories, e.g., cat, building, human face, and car. Images upscaled by GLEAN show clear improvements in terms of fidelity and texture faithfulness in comparison to existing methods.
We present a new way to exploit pre-trained GANs for the task of large-scale super-resolution, up to 64× upscaling factor.
The Framework
GLEAN
In contrast to existing works that optimize only the latent vectors of the generator (i.e. StyleGAN), GLEAN proposes to use the generator as a dictionary and condition it on the convolutional features provided by the encoder. In this way, the generator receives additional guidance on the local structures, producing results with high fidelity and quality.
Large Factor
Super-Resolution
GLEAN generates perceptually convincing images resembling the ground-truth for up to 64× upscaling. GLEAN can be applied to various categories by switching between StyleGANs trained on different categories.
Image
Retouching
In interactive image retouching, a perfect output typically requires tedious and precise editing steps. As a result, artifacts are common in the outputs, especially those from amateur retouching. Thanks to the capability of GLEAN in producing high quality and fidelity images, we can downsample an edit and upsample it using GLEAN to eliminate the blurry region and generate a coherent output with natural textures.
Real-World
Examples
Paper
Citation
@InProceedings{chan2021glean,
author = {Chan, Kelvin CK and Wang, Xintao and Xu, Xiangyu and Gu, Jinwei and Loy, Chen Change},
title = {GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2021}
}
Related
Projects
-
Towards Robust Blind Face Restoration with Codebook Lookup Transformer
S. Zhou, K. C. K. Chan, C. Li, C. C. Loy
in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
[arXiv] [Project Page] [Demo] -
GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond
K. C. K. Chan, X. Wang, X. Xu, J. Gu, C. C. Loy
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 (TPAMI)
[DOI] [arXiv] [Project Page] -
Investigating Trade-offs in Real-World Video Super-Resolution
K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Project Page] -
BasicVSR++: Effective Video Super-Resolution via Enhanced Propagation and Alignment
K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Project Page] -
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
K. C. K. Chan, X. Wang, K. Yu, C. Dong, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
[PDF] [Supplementary Material] [arXiv] [Project Page] -
Understanding Deformable Alignment in Video Super-Resolution
K. C. K. Chan, X. Wang, K. Yu, C. Dong, C. C. Loy
in Proceedings of AAAI Conference on Artificial Intelligence, 2021 (AAAI)
[arXiv] [Project Page]