Electro-optic sampling has emerged as a new quantum technique enabling measurements of electric field fluctuations on subcycle time scales. In a second-order nonlinear material, the fluctuations of a terahertz field are imprinted onto the polarization properties of an ultrashort probe pulse in the near infrared. The statistics of this time-domain signal are calculated, incorporating the quantum nature of the involved electric fields right from the beginning. A microscopic quantum theory of the electro-optic process is developed adopting an ensemble of noninteracting three-level systems as a model for the nonlinear material. It is found that the response of the nonlinear medium can be separated into a conventional part, which is exploited also in sampling of coherent amplitudes, and quantum contributions, which are independent of the state of the terahertz input. Interactions between the three-level systems which are mediated by terahertz vacuum fluctuations are causing this quantum response. Conditions under which the classical response serves as a good approximation of the electro-optic process are also determined and how the statistics of the sampled terahertz field can be reconstructed from the electro-optic signal is demonstrated. In a complementary regime, electro-optic sampling can serve as a spectroscopic tool to study the pure quantum susceptibilities of matter.