Millimeter-wave (mmWave) communication with a large bandwidth can result in a significantly improved data rate in wireless communications. To overcome high path-loss in the mmWave frequency band, beamforming technology is necessary. Especially, there has been widespread interest in development of hybrid beamforming (HB) technologies, in view of reducing cost and power consumption in massive multiple input multiple output (MIMO) systems. Some of existing researches on HB algorithms assumed perfect channel state information (CSI) and the others used beam training process in case of assuming imperfect CSI. When beam training process is used, enough beam training has to be conducted to achieve sufficient system performance in massive MIMO systems, which results in significant training overhead. Thus, it is necessary to reduce beam training complexity. Compared to state-of-the-art technology, we propose a multi-user HB system using codebooks based on a deep neural network (DNN) in this paper. In our proposed scheme, beam codewords for the base station (BS) and all users can be inferred using limited beam training in cases when the channel state information (CSI) is unknown. In order to apply the proposed scheme to situations where the CSI is unknown, reference radio frequency (RF) beamformers were introduced. Also, the proposed DNN structure is designed considering introduced reference RF beamformers. By using the proposed DNN with reference RF beamformers, the proposed system can inferred optimal beam codewords with limited beam training. Results obtained from simulations indicate that the proposed scheme can achieve almost the same performance as a conventional scheme with less beam training complexity. We also show that the performances achieved by the proposed scheme are gradually increased as training epoch is increased, eventually converging to a steady-state value.