Trajectory optimization problem for hypersonic vehicle is addressed in this thesis. The engine of hypersonic vehicle is assumed as a dual-mode scramjet engine which can be operated as a ramjet and scramjet for wide range of flight Mach number. Boost-skipping trajectory was proposed for maximization of hypersonic vehicle, and based on this trajectory, flight mode of dual-mode scramjet is divided into three modes: ram mode, scram mode, zero thrust mode. Hypersonic vehicle was modelled with consideration of changes of physical quantities over mode transition. To deal with discrete mode changes as well as continuous control, hybrid optimal control method is applied to this problem. Simulation results demonstrate that the optimized trajectory with hybrid control has better performance compared to rule-based mode transition trajectory. Also, a vehicle which imitates the characteristics of dual-mode scramjet vehicle is implemented to optimized the trajectory. The results suggest that the hybrid optimal control can be applied to the trajectory optimization of a dual-mode scramjet vehicle considering the mode transition.