GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields

📄 Paper

🤗 HuggingFace

🗓 Date: 9 Jun 2023

Main idea

• Motivation: NeRF models are highly dependent on carefully collected input data and properly chosen regularization strategies and may contain various artifacts during rendering for novel scenes.
• Solution: GANeRF - use GANs idea to improve the NeRF quality in challenging real-world scenarios. The key idea is to leverage an adversarial loss formulation end-to-end to introduce additional rendering constraints from a per-scene 2D discriminator.

Pipeline

The main idea is to use adversarial training to update and improve 3D representation. Additionally, a 2D generator is used to refine NeRF renderings based on feedback from a second discriminator.

In simple words, the authors use patches from original and rendered images to learn the first discriminator and NeRF model.

The final loss to train such a model can be presented as following:

where

Additionally, to improve the quality of rendered images the authors use 2D conditioned generator which takes as an input a NeRF rendering and an auxiliary random vector z and produces a cleaned image, thus serving as a stochastic denoiser.

Implementation details

Datasets:

• ScanNet++ (From the ScanNet++ dataset, the authors evaluate on five indoor scenes consisting of office, lab and apartment environment)
• Tanks and Temple (Auditorium, Ballroom, Courtroom, and Museum)

Metrics: PSNR, SSIM, LPIPS, KID scores

Compared with: Mip-NeRF 360, Instant NGP, Nerfacto, Nerfacto + pix2pix

Pros and cons

• Pros: improved quality and details of generated images.
• Limitations: patch discriminator is trained per scene.

Results