Image translation and manipulation with disentangled representations

Abstract

This paper proposes various methods to improve controllability during the generation process by disentangling the representation in the generation process. The study presents five methodologies. Firstly, by introducing new structural elements to StyleGAN, we divided the generation process into spatially invariant style control and spatial content control parts, thus achieving improved generation controllability. Secondly, a new methodology for image style transfer using text conditions was proposed, allowing the change of image texture information according to the given text condition while maintaining the structural components of the image. Thirdly, we proposed a new methodology for fine-tuning StyleGAN, enabling the generation of a new model capable of translating the images into the style of a given single-shot target image. Fourthly, we proposed a novel image translation strategy using a diffusion-based generative model that maintains structural content information during the sampling process while only changing style information. Lastly, leveraging the most recent text-to-image diffusion model, we extended the advantages of representation disentanglement, which was the goal of this research, to other tasks by combining it with image translation and concept personalization tasks.