HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement

Code

• github.com/isl-org/photorealismenhancement
  OfficialIn paper★ 1,333
• github.com/nvidia-cosmos/cosmos-transfer1
  OfficialIn paper★ 786
• github.com/junlinhan/dclgan
  OfficialIn paper★ 179
• github.com/stefanos50/regen
  OfficialIn paper★ 18
• github.com/stefanos50/hyper-gan
  OfficialIn paper★ 5

Abstract

Generative models are widely employed to enhance the photorealism of visual synthetic data for training computer vision algorithms. However, they often introduce visual artifacts that degrade the accuracy of these algorithms and require high computational resources, limiting their applicability in real-time training or evaluation scenarios. In this paper, we propose Hybrid Patch Enhanced Realism Generative Adversarial Network (HyPER-GAN), a lightweight image-to-image translation method based on a U-Net-style generator designed for real-time inference. The model is trained using paired synthetic and photorealism-enhanced images, complemented by a hybrid training strategy that incorporates matched patches from real-world images to improve visual realism and semantic consistency. Experimental results demonstrate that HyPER-GAN outperforms state-of-the-art lightweight paired image-to-image translation methods in terms of inference latency, visual realism, and semantic robustness. Moreover, it is illustrated that the proposed hybrid training strategy indeed improves visual quality and semantic consistency compared to training the model solely with paired synthetic and photorealism-enhanced images. Code and pretrained models are publicly available for download at: https://github.com/stefanos50/HyPER-GAN

Reproductions