Sketch Your Own GAN

Sheng-Yu Wang¹

David Bau²

Jun-Yan Zhu¹

¹CMU

²MIT CSAIL

Code [GitHub]

ICCV 2021 [Paper]

Our method takes in one or a few hand-drawn sketches and customizes an off-the-shelf GAN to match the input sketch. The same noise z is used for the pre-trained and customized models. While our new model changes an object’s shape and pose, other visual cues such as color, texture, background, are faithfully preserved after the modification.

Abstract

Can a user create a deep generative model by sketching a single example? Traditionally, creating a GAN model has required the collection of a large-scale dataset of exemplars and specialized knowledge in deep learning. In contrast, sketching is possibly the most universally accessible way to convey a visual concept. In this work, we present a method, GAN Sketching, for rewriting GANs with one or more sketches, to make GANs training easier for novice users. In particular, we change the weights of an original GAN model according to user sketches. We encourage the model’s output to match the user sketches through a cross-domain adversarial loss. Furthermore, we explore different regularization methods to preserve the original model's diversity and image quality. Experiments have shown that our method can mold GANs to match shapes and poses specified by sketches while maintaining realism and diversity. Finally, we demonstrate a few applications of the resulting GAN, including latent space interpolation and image editing.

Video

[Slides (30.3 MB)]

Paper

Sheng-Yu Wang, David Bau, Jun-Yan Zhu.
Sketch Your Own GAN.
In ICCV, 2021. (Paper)

[Bibtex]

Results

Our method can customize a pre-trained GAN to match input sketches.

Interpolation using our customized models. Latent space interpolation is smooth with our customized models.

Image 1

Interoplation

Image 2

Image editing using our customized models. (a) Given a real image, (b) we project it to the original model's noise z using Huh et al. (c) We feed the projected z to the standing cat model trained on sketches. (d) we edit the image with `add fur` operation using GANSpace.

We can interpolate between the customized model by interpolating the W-latent space.

Model 1

Interoplation in W-latent space

Model 2

We observe similar effect by interpolating the model weights directly.

Model 1

Interoplation in the model weight space

Model 2

Each video shows uncurated samples generated from a model trained on one or more sketch inputs. An example sketch input is shown.

Failure cases

Our method does not work well when sketch inputs depicts a complex pose, or when the sketches are drawn with a distinctive style. The generated images does not match the shapes and poses as faithfully as the previous models.

Related works

R. Gal, O. Patashnik, H. Maron, A. Bermano, G. Chechik, D. Cohen-Or. "StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators.". In ArXiv. (concurrent work)

D. Bau, S. Liu, T. Wang, J.-Y. Zhu, A. Torralba. "Rewriting a Deep Generative Model". In ECCV 2020.

Y. Wang, A. Gonzalez-Garcia, D. Berga, L. Herranz, F. S. Khan, J. van de Weijer. "MineGAN: effective knowledge transfer from GANs to target domains with few images". In CVPR 2020.

M. Eitz, J. Hays, M. Alexa. "How Do Humans Sketch Objects?". In SIGGRAPH 2012.

Acknowledgements

We thank Nupur Kumari and Yufei Ye for proof-reading the drafts. We are also grateful for helpful discussions from Gaurav Parmar, Kangle Deng, Nupur Kumari, Andrew Liu, Richard Zhang, and Eli Shechtman. We are grateful for the support of the Naver Corporation, DARPA SAIL-ON HR0011-20-C-0022 (to DB), and Signify Lighting Research. Website template is from Colorful Colorization.