Recently, convolutional neural network(CNN) has shown remarkable performance in the field of computer vision thanks to the availability of large-scale dataset. With its extremely high-level feature extraction capabilities, CNN has been expected to resolve automatic target recognition(ATR) problems. Since the automatic tareget recogntion(ATR) data is military-purpose data, it has a limited amount of labeled data, which is a problem in learning deep CNN. Thus, previous ATR methods have tried to supplement the data using simulated data or other available data. However, most of them are homogeneous sensor data rather than heterogeneous sensor due to distinctly different charateristics even though they have abundant knowledge to train CNN. To address these issues, we propose a transfer learning-based framework that can teach ATR algorithms using heterogeneous sensor data. As verification data of the method, we use unlabeled infrared(IR) data as target data and labeled electro optical(EO) data as source data. The verification results demonstrate that the transfer learning scheme can train IR-ATR CNN to learn sensor invariant features of the target with labeled heterogeneous sensor data i.e. EO, which is not possible with normal supervised learning.