We describe a new method for monitoring the "in-band" optical signal-to-noise ratio (OSNR) of differential phase-shift-keying (DPSK) and differential quadrature phase-shift-keying (DQPSK) signals. This technique estimates the OSNR by analyzing the radio frequency spectrum obtained by the self-heterodyne detection. The results show that the performance of the proposed technique is not sensitive to the effects of chromatic dispersion and polarization-mode dispersion. In addition, this technique can be used for monitoring the polarization-scrambled signals. For a demonstration, we measure the OSNR of the polarization-scrambled 10-Gb/s DPSK and 20-Gb/s DQPSK signals in a 640-km-long transmission link.