In this paper, we extend cascaded rectangle features based face detection to facial expression. New types of rectangle features that are suitable for facial expression recognition (FER) are proposed, We choose those as rectangle features for FER in a 3 x 3 matrix form, using a variant of AdaBoost. In addition, the FER system constituted with the proposed rectangle feature set is compared to that with a typical rectangle feature set with regard to its capacity. Simulation and experimental results indicate that the proposed features show better representation and classification in FER. (c) 2008 Elsevier B.V. All rights reserved.