Similarity-assisted variational autoencoder for nonlinear dimension reduction with application to single-cell RNA sequencing data

Abstract

Deep generative models naturally become nonlinear dimension reduction tools to visualize large-scale datasets for revealing latent grouping patterns or identifying outliers. Variational autoencoder (VAE) is a deep generative method equipped with encoder/decoder structures. However, the VAE tends not to show the grouping pattern clearly without additional annotation information. On the other hand, similarity-based dimension reduction methods such as t-SNE or UMAP present a clear grouping pattern even though these methods do not have encoder/decoder structures. To bridge this gap, we propose a new approach that adopts similarity information in the VAE framework. Our proposed method finds lower dimensional representations with clear grouping structures while keeping the encoder/decoder structures in the model. For biological applications, it is crucial to adjust for covariate information such as batch or doner information to find biologically meaningful groups. We then extend our approach to a conditional VAE (CVAE) to incorporate the covariate information in the dimension reduction step. Our method shows great performance on both synthetic and real-world datasets.