In this paper, nonlinear H∞ optimal design of impedance controllers is proposed based on the nonlinear robust internal-loop compensator (NRIC) framework. Simply adding PD-type auxiliary input to the original control law, the robust performance and the robust stability are achieved. Nonlinear H∞ optimality is guaranteed by solving Hamilton-Jacobi-Isaacs (HJI) equation and the disturbance input-to-state stability (ISS) is guaranteed by finding ISS-Lyapunov function. Moreover, it is shown that the proposed method preserves the passivity of the impedance controllers. The proposed method can be applied to various types of impedance controllers in a unified way. Through simulations and experimental studies, the proposed method is verified.