Many researchers have recently attempted to predict the emotions in conversations, which is essential to developing a human-like chatbot system. However, it is challenging to build a desirable emotion recognition model due to the emotion-labeled data scarcity, especially in Korean. A previous study presented a distant supervision-based annotation procedure with the use of emotion lexicons. However, this procedure has two potential problems: (1) it is too dependent on the emotion lexicons; (2) it is hard to capture long-range contextual information during the conversation. This paper addresses two problems by utilizing a pre-trained deep learning model, which has achieved good performance on several dialogue emotion datasets, as an annotator. Experiments demonstrate that the pre-trained model is more desirable to create emotion labels on each utterance during the conversation.