This article proposes an efficient approach for solving three-dimensional (3D) topology optimization problem. In this approach, the number of design variables in optimization as well as the number of degrees of freedom in structural response analysis can be reduced significantly. This is accomplished through the use of scaled boundary finite element method (SBFEM) for structural analysis under the moving morphable component (MMC)-based topology optimization framework. In the proposed method, accurate response analysis in the boundary region dictates the accuracy of the entire analysis. In this regard, an adaptive refinement scheme is developed where the refined mesh is only used in the boundary region while relating coarse mesh is used away from the boundary. Numerical examples demonstrate that the computational efficiency of 3D topology optimization can be improved effectively by the proposed approach.