End-to-End Unsupervised Sketch to Image Generation

A freehand sketch is geometrically distorted and lacks colors, textures, and other visual details. This leads to challenges in freehand sketch-to-image generation tasks. Many existing works take a multi-stage strategy to generate images from sketches, and they depend on the shape of the input sketch badly and ignore the shape adjustment of the generated image. More importantly, the generated images are of low quality with distorted textures and colors, and shape deformation. In order to overcome the above challenges, we propose an end-to-end method to accomplish the freehand sketch-to-image task, and the proposed architecture is based on an unsupervised network. Our key insight is to propose a shape discriminator to provide shape constraints for the generated image. Besides, we introduce Image-Global Attention(IGA) and Focal Frequency Loss(FFL). IGA and FFL mainly focuses on the whole image and every patch of an image, respectively. We also extend a new dataset called NewGiraffe based on the giraffe class of SketchyCOCO, and our approach is validated on two datasets: Shoes and NewGiraffe. Through qualitative and quantitative results, we demonstrate our method's ability to generate realistic images from freehand sketches.