An Energy-Efficient GAN Accelerator with On-chip Training for Domain Specific Optimization

Abstract

Generative Adversarial Networks (GAN) consists of multiple deep neural networks cooperating and competing each other. Due to their complex architectures, training GANs requires huge computations and external memory accesses. However, retraining GANs with user-specific data is critical on mobile devices because the pretrained model outputs distorted images under user-specific conditions. This paper proposes an FPGA-based GAN training accelerator to enable energy-efficient domain specific optimization of GAN with user's local data. A SELRET(Selective-Layer Retraining) is proposed to reduce the computation by 69% through selecting layers that are effective in enhancing the quality of the output. Moreover, ROLIN (Reordering Layers for Instance Normalization) is proposed to reduce the external memory accesses of intermediate data. The design is implemented on Intel's Cyclone V, showing 51 GFLOPS peak performance with 10.04 GFLOPS/W energy-efficiency for face modification of 256x256 image.