This paper proposes a new formation guidance law for multiple UAVs in a three-dimensional space. The proposed formation guidance law produces a velocity command vector of a wingman to form a prescribed formation shape at a specified formation time. The guidance command ensures that the time derivative of a Lyapunov function defined as the square of zero-effort-miss between a wingman and a desired formation position is negative definite during the flight. The velocity command vector is then transformed to typical autopilot inputs of a UAV: speed, flight path angle and heading angle commands. The performance of the proposed law is evaluated using a full nonlinear 6-DOF UAV model in the cases of formation shaping, formation keeping, and formation re-shaping when the leader is accelerating. Simulation results show that the proposed law maintains the formation keeping well and precisely achieves the given shape of formation even when the leader is accelerating.