In recent years, considerable efforts based on convolutional neural networks have been devoted to age estimation from face images. Among them, classification-based approaches have shown promising results, but there has been little investigation of age differences and ordinal age information. In this paper, we propose a ranking objective with two novel schemes jointly performed with an age classification objective to take ordinal age labels into account. We first introduce relative triplet sampling in which a set of triplets is constructed considering the relative differences in ages. This also addresses the problem of having limited triplet candidates, that occurs in conventional triplet sampling. We then propose the scale-varying ranking constraint, which decides the importance of a relative triplet and adjusts a scale of gradients accordingly. Our adaptive ranking loss with relative sampling not only lowers the generalization error but ultimately has a meaningful performance improvement over the state-of-the-art methods on two well-known benchmarks.