A low-power, highly secure, always-on face recognition (FR) processor is required for security applications. In this brief, a branch net-based early stopping FR (BESF) processor is proposed to prevent adversarial attacks for high security and consume low power for always-on operation. It shows a recognition accuracy of 83.10% under the fast gradient signed method (FGSM), and 71.97% under the projected gradient descent (PGD) attack. The clock-gating of the BESF processor reduces the average power consumption by 30.85%. The unified pointwise and depthwise convolution processing element adopts layer-fusion to reduce the external memory access by 88.0%. Furthermore, noise injection layers are inserted between every bottleneck layer to further reduce the FGSM and PGD attack success rate by 9.29% and 20.0%, respectively. Implemented with a 65 nm CMOS process with a 3.0 mm $\times $ 3.0 mm area, the processor consumes 0.22-0.89 mW power at 1 fps and shows 95.5% FR accuracy in the Labeled Faces in the Wild dataset.