In this paper, we propose a method for designing an optimal filter for QAM-FBMC system in the presence of the residual CFO. We model the QAM-FBMC system with residual CFO, and describe the optimization conditions for the prototype filter design. By using these criteria, we formulate the prototype filter design problem for the QAM-FBMC system with the residual CFO, and present the designed filter coefficients. Finally, simulation results show the performance comparison of the proposed prototype filters, and we confirm that the designed filters have the performances of the design intents.