Recent experiments point to a variety of intermetallic systems which exhibit exotic quadrupolar orders driven by the Kondo coupling between conduction electrons and localized quadrupolar degrees of freedom. Using a Luttinger k center dot p Hamiltonian for the conduction electrons, we study the impact of such quadrupolar order on their energies and wave functions. We discover that such quadrupolar orders can induce a nontrivial Berry curvature for the conduction electron bands, leading to a nonvanishing optical gyrotropic effect. We estimate the magnitude of the gyrotropic response in a candidate quadrupolar material, PrPb3, and discuss the resulting Faraday rotation in thin films.