PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

Code

• github.com/adamian98/pulse
  Officialpytorch★ 8,015
• github.com/tg-bomze/Face-Depixelizer
  none★ 2,563
• github.com/krantirk/Self-Supervised-photo
  pytorch★ 84
• github.com/mlaskowski17/Pulse
  pytorch★ 13
• github.com/marcin-laskowski/Pulse
  pytorch★ 13
• github.com/kainoj/keep-pulse
  pytorch★ 4
• github.com/Martion-z/Paddle-PULSE
  paddle★ 3
• github.com/ytsheng/sketch2cat-pulse
  pytorch★ 3
• github.com/kingsj0405/pulse-any-image
  pytorch★ 2
• github.com/Martion-z/pytorch-Pulse
  pytorch★ 1

Abstract

The primary aim of single-image super-resolution is to construct high-resolution (HR) images from corresponding low-resolution (LR) inputs. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present an algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require supervised training on databases of LR-HR image pairs). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee realistic outputs. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show proof of concept of our approach in the domain of face super-resolution (i.e., face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously possible.

Tasks

Benchmark Results

Reproductions