In this work, we develop geometric multigrid algorithms for the immersed finite element methods for elliptic problems with interface (Chou et al. Adv. Comput. Math. 33, 149-168 2010; Kwak and Lee, Int. J. Pure Appl. Math. 104, 471-494 2015; Li et al. Numer. Math. 96, 61-98 2003, 2004; Lin et al. SIAM J. Numer. Anal. 53, 1121-1144 2015). We need to design the transfer operators between levels carefully, since the residuals of finer grid problems do not satisfy the flux condition once projected onto coarser grids. Hence, we have to modify the projected residuals so that the flux conditions are satisfied. Similarly, the correction has to be modified after prolongation. Two algorithms are suggested: one for finite element spaces having vertex degrees of freedom and the other for edge average degrees of freedom. For the second case, we use the idea of conforming subspace correction used for P-1 nonconforming case (Lee 1993). Numerical experiments show the optimal scalability in terms of number of arithmetic operations, i.e., O(N) for V-cycle and CG algorithms preconditioned with V-cycle. In V-cycle, we used only one Gauss-Seidel smoothing. The CPU times are also reported.