How GANs’ Objective Functions Evolved: From JS Divergence to Modern Variants

Introduction

Many think algorithms and programming are far from art, but they contain a highly creative world built on logic. Generative Adversarial Networks (GANs) exemplify this creativity. GANs combine a generative model that aims to approximate a joint probability distribution with an adversarial training scheme, a concept praised by Yann LeCun as one of the most interesting ideas in machine learning over the past decade. Since their 2014 proposal, GANs have expanded into image, audio, text, poetry, and more.

Question

Briefly describe the evolution of GANs' objective functions.

Analysis and Answer

The goal of a generative model is to make the generated distribution as close as possible to the real distribution, so reducing the divergence between the two is key. The original GANs used Jensen‑Shannon (JS) divergence, which can cause gradient vanishing. Recent methods replace JS with other divergence measures or distance metrics to improve performance. The main categories are:

f‑divergence based GANs (see Table 1).

Integral Probability Metric (IPM) based GANs.

GANs with auxiliary objectives such as reconstruction or classification losses.

IPM‑based objectives avoid the estimation difficulties of f‑divergences in high‑dimensional data, providing stable convergence even when the supports of the two distributions do not overlap.

Auxiliary objectives can improve training stability or add new capabilities. Reconstruction losses encourage generated samples to resemble real ones, useful for image‑to‑image translation. Adding classification losses enables semi‑supervised learning or style transfer, often by attaching a cross‑entropy term to the discriminator.

Table 1: GANs based on f‑divergence (illustrated above).

References

Goodfellow I, Pouget‑Abadie J, Mirza M, et al. Generative adversarial nets. NIPS 2014.

Zhu J‑Y, Park T, Isola P, et al. Unpaired image‑to‑image translation using cycle‑consistent adversarial networks. CVPR 2017.

Zhao J, Mathieu M, LeCun Y. Energy‑based generative adversarial network. arXiv 2016.

Arjovsky M, Chintala S, Bottou L. Wasserstein GAN. arXiv 2017.

Mroueh Y, Sercu T, Goel V. McGAN: Mean and covariance feature matching GAN. arXiv 2017.

Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of Wasserstein GANs. NIPS 2017.

Wei X, Gong B, Liu Z, et al. Improving the improved training of Wasserstein GANs: A consistency term and its dual effect. arXiv 2018.

Kodali N, Abernethy J, Hays J, et al. On convergence and stability of GANs. arXiv 2017.

Lim J H, Ye J C. Geometric GAN. arXiv 2017.

Zhang H, Goodfellow I, Metaxas D, et al. Self‑attention generative adversarial networks. arXiv 2018.

Brock A, Donahue J, Simonyan K. Large scale GAN training for high fidelity natural image synthesis. arXiv 2018.