This paper presents a trajectory tracking controller for multi-rotor UAVs to improve their flight performance in the presence of various uncertainties. The proposed tracking system consists of a velocity guidance law based on relative distance and an L1 adaptive augmentation loop for tracking velocity commands. In the proposed structure, the desired velocity generated by the guidance law is the reference value for the adaptive controller for accurate path tracking. In the guidance law, the desired acceleration is generated based on the relative distance and its derivatives, and the velocity command of the inner control loop is then calculated. The L1 augmentation loop compensates the linear controller to guarantee the flight performance parameters such as tracking accuracy in the presence of uncertainties, which include aerodynamic disturbances, modeling error, outdoor environmental factors, and flight dynamics changes. The proposed controller was validated in actual flight tests to successfully demonstrate its capabilities using a quad-rotor UAV.