This paper proposes a real-time footstep planning framework based on the A∗ footstep planning algorithm. The proposed footstep planner utilizes an energy consumption-based cost function to generate energy efficient navigation plan. To increase the computational efficiency, the cost-to-go heuristic function, which approximates the remaining cost to reach the goal, is modeled more precisely by considering the angle difference between the robot and the goal position. Furthermore, an efficient dual-level feasibility check is done to avoid occlusion with the environment while stepping over small objects if possible. Finally, the proposed footstep planning algorithm is integrated with the mapping algorithm and the walking controller to solve the humanoid navigation problem and validated in simulation and real-world experiments.