This paper considers a deep Generative Adversarial Networks (GAN) based method referred to as the Perception-Enhanced Super-Resolution (PESR) for Single Image Super Resolution (SISR) that enhances the perceptual quality of the reconstructed images by considering the following three issues: (1) ease GAN training by replacing an absolute with a relativistic discriminator, (2) include in the loss function a mechanism to emphasize difficult training samples which are generally rich in texture and (3) provide a flexible quality control scheme at test time to trade-off between perception and fidelity. Based on extensive experiments on six benchmark datasets, PESR outperforms recent state-of-the-art SISR methods in terms of perceptual quality. The code is available at https://github.com/thangvubk/PESR.