Disentangling and diversifying text-driven image manipulation via StyleGAN generator

Abstract

With the advent of multi-modal representation models such as CLIP, text-driven style transfer has recently been in the spotlight for that it is more intuitive and directly interpretable. Nevertheless, not much attention has been paid to an unexpected issue that are newly emerging for this new type of manipulation model. In this paper, we propose a novel approach to bypass the entanglement of text guidance with integrated cross-modal information. In addition, we raise the issue that the state-of-the-art language-guided image manipulation in fact cannot generate sufficiently diverse samples due to the rigidity of text guidance. To address this issue, we propose a simple yet effective method, Cross-Modal embedding Surgery (CosMoS), which automatically searches for the desired set of semantics from image and text, then combines the selected subset for source-preserved and multiple-styled results. We show the validity of our method that generates diversified manipulation results in the human and animal face domain.