The influence of measurement back action on electro-optic sampling of electromagnetic quantum fluctuations is investigated. Based on a cascaded treatment of the nonlinear interaction between a near-infrared coherent probe and the mid-infrared vacuum, we account for the generated electric-field contributions that lead to detectable back action. Specifically, we theoretically address two realistic setups, exploiting one or two probe beams for the nonlinear interaction with the quantum vacuum, respectively. The setup parameters at which back action starts to considerably contaminate the measured noise profiles are determined. We find that back action starts to detrimentally affect the signal once the fluctuations due to the coupling to the mid-infrared vacuum become comparable to the base shot noise. Due to the vacuum fluctuations entering at the beam splitter, the shot noise of two incoming probe pulses in different channels is uncorrelated. Therefore, even when the base shot noise dominates the output of the experiment, it does not contribute to the correlation signal itself. However, we find that further contributions due to nonlinear shot-noise enhancement are still present. Ultimately, a regime in which electro-optic sampling of quantum fields can be considered as effectively back-action free is found.