Positron emission tomography (PET) images usually suffer from incorrect quantification of the radioactive uptake of small lesions due to low spatial resolution. To improve the spatial resolution, we previously proposed super-resolution (SR) algorithms based on wobble scanning. The proposed algorithms, however, require mechanical motion of the patient bed or a system gantry for wobble scanning. In this paper, we propose a framework for SR reconstruction of 3D PET images based on the use of respiratory motion rather than wobble motion. As in the conventional protocol of PET imaging, gated list-mode PET data are acquired in a free breathing condition. In addition, we acquire two low-dose CT images in a breath-hold manner at exhale and inhale phases, without increasing the radiation burden to a patient. Using the two low-dose CT images, we estimate the 4D motion vector field (MVF) and correspondingly generate a virtual 4D CT image that are matched to the 4D PET image. The 3D CT images have much better spatial resolution than PET images and therefore the corresponding estimated 3D MVFs can be considered reliable for PET SR reconstruction. We then estimate space-variant point spread functions (PSFs) in the imaging field of view using a minimum number of PSFs obtained through Monte-Carlo simulations. Finally, SR reconstruction is performed by incorporating the estimated 3D MVFs and space-variant PSFs. In the SR reconstruction, to avoid the resolution degradation in the evenly spaced parallel-beam rebinning and to reduce the computational time on the graphics processing unit, we introduce a parallel-friendly spanned line of response reconstruction technique based on fan-beam reordering. The proposed framework is evaluated via Monte-Carlo simulations with the digital XCAT phantom and via experiments with several patient datasets including moving lung lesions. Both the simulation and experiment results show that the proposed framework provides much clearer organ boundaries as well as more accurate quantitative lesion information than the conventional methods, with a considerable reduction of computational time.