As many as eight multiple plasma bullets produced at atmospheric pressure were observed in one voltage period in a capillary helium dielectric barrier plasma jet. We found that the number of the bullets strongly depends on the nitrogen fraction added to the helium supply gas. Using optical emission spectroscopy and ionization rate calculation, this study demonstrates that nitrogen gas plays an important role in the generation and dynamics of multiple plasma bullets through Penning ionization of nitrogen by helium metastables.