This paper presents the improved isolation property of the signals among transmitter and receiver antennas. The separation wall is laid at the center of the antennas to improve the isolation level between them. The introduced separation wall has serrated edges mounted on three sides, i.e., top, left and right sides. These mounted serrated edges are implemented to reduce the diffraction which may occur due to the linear edge of the wall. The Fresnel diffraction problem has been solved using analytical method in order to get the optimized structure of the serration. The Fresnel diffraction patterns due to different sizes of the serration are obtained, and their relative powers are compared to each other. The implemented antenna system with the serration wall is composed of corrugated feed horn, orthogonal mode transducer, and offset dual-reflector parabolic antennas. The effect of serration is well demonstrated by the measurement of isolation level of the antenna system. The measured results show that the serrated edges enhanced the isolation property among transmitter and receiver antennas.