First-order reliability method (FORM) has been mostly utilized for solving reliability-based design optimization (RBDO) problems efficiently. However, second-order reliability method (SORM) is required in order to estimate a probability of failure accurately in highly nonlinear performance functions. Despite accuracy of SORM, its application to RBDO is quite challenging due to unaffordable numerical burden incurred by a Hessian calculation. For reducing the numerical efforts, a quasi-Newton approach to approximate the Hessian is introduced in this study instead of calculating the true Hessian. The proposed SORM with the approximated Hessian requires computations only used in FORM, leading to very efficient and accurate reliability analysis. The proposed SORM also utilizes a generalized chi-squared distribution in order to achieve better accuracy. Furthermore, SORM-based inverse reliability method is proposed in this study. An accurate reliability index corresponding to a target probability of failure is updated using the proposed SORM. Two approaches in terms of finding an accurate most probable point using the updated reliability index are proposed. The proposed SORM-based inverse analysis is then extended to RBDO in order to obtain a reliability-based optimum design satisfying probabilistic constraints more accurately even for a highly nonlinear system. The numerical study results show that the proposed reliability analysis and RBDO achieve efficiency of FORM and accuracy of SORM at the same time.