We address the application of the invertible neural network to the unsupervised denoising of low dose CT image data. Recently, CycleGAN was shown to provide high-performance, ultra-fast denoising for low dose X-ray computed tomography (CT) without the need for a paired training dataset. Although this was possible thanks to cycle consistency, CycleGAN requires two generators and two discriminators to enforce cycle consistency, demanding significant GPU resources and technical skills for training. To solve this problem, here we present a novel cycle-free CycleGAN architecture, which consists of a single generator and a discriminator but still guarantees the cycle consistency. The main innovation comes from the observation that the use of an invertible generator automatically fulfills the cycle consistency condition and eliminates the additional discriminator in the CycleGAN formulation. Extensive experiments using various levels of low dose CT images confirm that our method can improve denoising performance using only 24\% of learnable parameters, twice faster training time, and showed better trade-offs between the complexity versus reconstruction quality, compared to the existing methods.