Object detection became one of the major fields in computer vision. In object detection, object classification and object localization tasks are conducted. Previous deep learning-based object detection networks perform with feature maps generated by completely shared networks. However, object classification focuses on the most discriminative object part of the feature map. Whereas, object localization requires a feature map that is focused on the entire area of the object. In this paper, we propose a novel object detection network by considering the difference between the two tasks. The proposed deep learning-based network mainly consists of two parts; 1) Attention network part where task-specific attention maps are generated, 2) Layer separation part where layers for estimating two tasks are separated. Comprehensive experimental results based on PASCAL VOC dataset and MS COCO dataset showed that proposed object detection network outperformed the state-of-the-art methods.