Spectral computed tomography (CT) is a promising technique with the potential for improving lesion detection, tissue characterization, and material decomposition. In this paper, we are interested in kVp switching-based spectral CT that alternates distinct kVp X-ray transmissions during gantry rotation. This system can acquire multiple X-ray energy transmissions without additional radiation dose. However, only sparse views are generated for each spectral measurement; and the spectra themselves are limited in number. To address these limitations, we propose a penalized maximum likelihood method using spectral patch-based low-rank penalty, which exploits the self-similarity of patches that are collected at the same position in spectral images. The main advantage is that the relatively small number of materials within each patch allows us to employ the low-rank penalty that is less sensitive to intensity changes while preserving edge directions. In our optimization formulation, the cost function consists of the Poisson log-likelihood for X-ray transmission and the nonconvex patch-based low-rank penalty. Since the original cost function is difficult to minimize directly, we propose an optimization method using separable quadratic surrogate and concave convex procedure algorithms for the log-likelihood and penalty terms, which results in an alternating minimization that provides a computational advantage because each subproblem can be solved independently. We performed computer simulations and a real experiment using a kVp switching-based spectral CT with sparse-view measurements, and compared the proposed method with conventional algorithms. We confirmed that the proposed method improves spectral images both qualitatively and quantitatively. Furthermore, our GPU implementation significantly reduces the computational cost.