We propose a user selection algorithm for the downlink multi-user millimeter wave systems. Our design aims to classify favorable users in terms of the achievable rate based on the channel state information. The phased-zero-forcing scheme is introduced to find the analog precoder gain easily under the multi-user environment. In order to maximize the overall sum-rate of selected users, we then establish an appropriate criterion, which quantifies the fidelity to each sum-rate maximization under interference condition. Some numerical results show that the performance of the proposed algorithm is very close to that of the conventional full search scheme. It is also observed that the complexity of proposed algorithm can be further reduced by setting the proper value of threshold, while it still exhibits almost the same performance as that of the full search.