Double compression (DC) involves compressing a video twice with the video codec. DC usually occurs when a compressed video is manipulated, because most videos are saved in the compression format after manipulating. Given this principle, if we can detect DC traces in videos, we can determine whether a video has been manipulated. Prior works have proposed methods for detecting DC of H.264 videos; however, they have limitations in that they can detect DC only for a limited dataset and set of compression parameters. To overcome these limitations, we propose a two-stream neural network that incorporates two components that analyze intra-coded frames and predictive frames. We explain why the proposed two-stream neural network can detect DC traces in videos more accurately than the prior method, and we demonstrate that the proposed network can detect DC videos that have various contents and parameters through extensive experiments.