This study presents an age and gender estimation system that considers ethnic difference in face images using a Convolutional Neural Network(CNN) and Support Vector Machine(SVM). Most age and gender estimation systems using face images are trained on ethnicity-biased databases. Therefore, these systems show limited performance on face images of ethnic groups occupying a small proportion of the training data. To resolve this problem, we propose an age and gender estimation system that considers the ethnic difference in face images. At the first stage of the system, the ethnicity of the facial image is determined by a CNN trained with manually collected face images of Asian and non-Asian celebrities. Then, one of the SVM classifiers is selected according to the ethnicity for the final age and gender estimation. We compared the proposed system with an estimation system that does not consider ethnic difference. The result shows improved performance for age estimation but no improvement for gender recognition.