The fundamental theory of Raman and infrared (IR) linewidth has been well established as the third-order lattice anharmonicity (three-phonon scattering). In this work, we use both rigorous density functional calculations and Raman experiments to find, surprisingly, that the fourth-order anharmonicity universally plays a significant or even dominant role over the third-order anharmonicity at room temperature, and more so at elevated temperatures, for a wide range of materials including diamond, Si, Ge, GaAs, boron arsenide (BAs), cubic silicon carbide (3C-SiC), and alpha-quartz. This is enabled by the large four-phonon scattering phase space of zone-center optical phonons. Raman measurements on BAs were conducted, and their linewidth verifies our predictions. The predicted infrared optical properties through the Lorentz oscillator model, after including four-phonon scattering, show much better agreement with experimental measurements than those three-phonon-based predictions. Our work advances the fundamental understanding of Raman and IR response and will broadly impact spectroscopy techniques and radiative transport.