In this paper, atomistic-level quantum mechanical simulations are performed for nanoscale field-effect transistors (FETs) with lateral or vertical heterojunction, within the non-equilibrium Green's function formalism. For efficient simulation of such heterostructure FETs, a novel approach is developed where the Green's functions are calculated by complementarily using the two algorithms of the recursive Green's function and the R-matrix. The R-matrix algorithm is extended to seamlessly combine the two methods on the open system and an algorithm for the electron correlation function based on the extended R-matrix algorithm is also developed. The proposed method significantly reduces simulation time, making rigorous atomistic simulations of heterojunction FETs possible. As an application, device simulations are performed for the germanane/InSe vertical tunneling FET (VTEET) modeled through the first-principles density functional theory. Our simulation results reveal that the germanane/InSe VTEET is a promising candidate for future low power applications.