We propose two rate selection methods in wireless multicast systems where an incremental redundancy type hybrid automatic repeat request (HARQ-IR) scheme is employed under Rayleigh block-fading channels. We aim at maximizing the long-term average transmission rate (LATR) while the maximum number of retransmissions is limited to L-1 and the probability that at least one user fails to decode a message within L (re)transmissions is limited to epsilon. This problem is non-convex and involves complicated integral expressions. To reduce the computational load, we formulate a simplified problem with approximated utility and constraint functions by using the Gaussian approximation.