Sparsity-aware autoencoder-based embedding for graph-structured data

Abstract

Finding low-dimensional embeddings of sparse high-dimensional data objects is very important in various fields such as recommendation, graph mining, and natural language processing. Recently, autoencoder (AE)-based embedding approaches have achieved state-of-the-art performance in many tasks, especially in top-$\kappa$ recommendation tasks with user embedding or node classification tasks with node embedding. However, we find that since many real-world data follow the power-law with respect to the data object sparsity, AE-based embedding severely suffers from a problem, which we call polarization, that dense data objects move away from sparse ones in an embedding space even if they are highly correlated. In this paper, we propose TRAP that leverages two-level regularizers to effectively alleviate this problem. (i) The “macroscopic regularizer” adds a regularization term in the loss function to generally prevent dense input objects to being distant from other sparse input objects. (ii) The “microscopic regularizer” introduces a new object-wise parameter to individually entice each object to correlated neighbor objects rather than uncorrelated ones. Importantly, TRAP is a meta-algorithm that can be easily coupled with existing AE-based embedding methods with a simple modification. In extensive experiments on two representative embedding tasks using six-real world datasets, TRAP boosted the performance of the state-of-the-art algorithms by up to 31.53% and 94.99% respectively.