Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation

Yasheng Sun*1 Qianyi Wu*2 Hang Zhou*3 Kaisiyuan Wang4 Tianshu Hu3 Chen-Chieh Liao1 Dongliang He3 Jingtuo Liu3 Errui Ding3 Jingdong Wang3 Shio Miyafuji1 Ziwei Liu5 Hideki Koike1

Abstract


Creating the photo-realistic version of people sketched portraits is useful to various entertainment purposes. Existing studies only generate portraits in the 2D plane with fixed views, making the results less vivid. In this paper, we present Stereoscopic Simplified Sketch-to-Portrait (SSSP), which explores the possibility of creating Stereoscopic 3D-aware portraits from simple contour sketches by involving 3D generative models. Our key insight is to design sketch-aware constraints that can fully exploit the prior knowledge of a tri-plane-based 3D-aware generative model. Specifically, our designed region-aware volume rendering strategy and global consistency constraint further enhance detail correspondences during sketch encoding. Moreover, in order to facilitate the usage of layman users, we propose a Contour-to-Sketch module with vector quantized representations, so that easily drawn contours can directly guide the generation of 3D portraits. Extensive comparisons show that our method generates high-quality results that match the sketch. Our usability study verifies that our system is greatly preferred by user.

Demo Video



Materials


Citation

@misc{https://doi.org/10.48550/arxiv.2302.06857,
  doi = {10.48550/ARXIV.2302.06857},
  url = {https://arxiv.org/abs/2302.06857},
  author = {Sun, Yasheng and Wu, Qianyi and Zhou, Hang and Wang, Kaisiyuan and Hu, Tianshu and Liao, Chen-Chieh and He, Dongliang and Liu, Jingtuo and Ding, Errui and Wang, Jingdong and Miyafuji, Shio and Liu, Ziwei and Koike, Hideki},
  title = {Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation},
  publisher = {arXiv},
  year = {2023},
  copyright = {Creative Commons Attribution 4.0 International}
}