We investigated the correlation between the impact toughness and microstructures of annealed Fe-8Mn-0.2C-3Al-1.3Si (wt.%) steel to identify the key microstructural feature determining the impact toughness of medium-Mn steel. The microstructural constituents were varied by changing the hot-rolling temperature in the range of 1000–1200 °C before intercritical annealing. The annealed steels exhibited a triplex-phase microstructure consisting of δ ferrite with coarse grains and an elongated structure along the rolling and transverse directions and nanolaminate α martensite plus γR retained austenite with ultrafine size. While the volume fraction of γR remained almost constant regardless of the hot-rolling temperature, the volume fraction of δ increased and that of α decreased with increase in the hot-rolling temperature. The average grain size for all phases increased with the hot-rolling temperature. The stability of γR decreased with the increase of the hot-rolling temperature owing to grain coarsening and a reduction in the Mn and C concentrations. A lower hot-rolling temperature resulted in improved impact toughness. We observed that deep parallel cracks formed and propagated along the δ interface decorated with Mn, ultimately causing a fracture. This result indicates that δ ferrite was the crucial factor determining the toughness among the existing phases, and the steels with a higher fraction of δ exhibited a lower impact toughness. The decrease of the retained austenite stability and the increase of the size of prior γ grains with increasing hot-rolling temperature were identified as other microstructural factors determining the impact toughness.