In this study, we propose phoxonic crystals with structural hierarchy to enhance the acousto-optic interaction. In previous studies, researchers investigated the interaction between light and sound which have same wavelength scales, so infrared interacts with high-frequency acoustic wave in GHz range. However, the light cannot interact strongly with the sound due to high acoustic attenuation coefficient which is proportional to the square of frequency. By using acoustic wave with low frequency, the acousto-optic interaction can be enhanced, but the wavelength of sound is far different from that of the infrared. Therefore, it is necessary to investigate the phoxonic crystals which can control waves with different wavelength. In this study, we propose the phoxonic crystals with structural hierarchy and a cavity. The proposed phoxonic crystals confine more optic waves at the cavity than conventional phoxonic crystals, so the acousto-optic interaction is enhanced by about 2.8 times. In order to confine more acoustic waves, we propose three-phase phoxonic crystals with structural hierarchy by using mathematical homogenization theory. The structures further enhance the confinement of sound and the acousto-optic interaction.