A distributed in-situ analysis method for large-scale scientific data

Abstract

Recently, a massive amount of data is generated in a wide range of scientific applications such as NASA's satellite, the large hadron collider, and large synoptic survey telescope. Most of scientific data follows the array model, and there are various kinds of standard array formats such as HDF, NetCDF, MDSplus, and ROOT. SciDB is the most well-known DBMS that stores the array-based scientific data and processes queries on it. SciDB is a distributed DBMS, and so, is scalable in terms of query performance. However, it has a severe drawback that takes a huge amount of time for loading a massive amount of scientific data into DBMS. That is, it is not scalable in terms of data loading. To overcome that problem, we propose a distributed in-situ analysis method that allows processing queries on raw scientific data in a distributed manner without explicit data loading. In detail, we propose the in-situ scan operator that scans necessary data of the array format and passes it to upper operators of the pipeline of a query plan. It also performs repartitioning during in-situ scanning, which is required for correct query results. Through experiments using real datasets, we have shown that the SciDB system using our method significantly outperforms the original SciDB system by orders of magnitude in terms of the performance of the first query. ? 2017 IEEE.