Anchor boxes act as potential object localization candidates allow single-stage detectors to achieve real-time performance, at the cost of localization accuracy when compared to state-of-the-art two-stage detectors. Therefore, correct selection of the scale and aspect ratio associated with an anchor box is crucial for detector performance. In this work, we propose a novel architecture called DANet for improving the localization performance of single-stage object detectors, while maintaining real-time inference. The proposed network achieves this by predicting (1) the combination of aspect ratio and scale per feature map based on object density and (2) localization confidence per anchor box. We evaluate the proposed network using the benchmark dataset. On the MS COCO dataset, DANet achieves 30.9% AP at 51.8 fps using ResNet-18 and 45.3% AP at 7.4 fps using ResNeXt-101. The code and models will be available at https://github.com/PS06/AnchorNet.