A Component-Based Architecture for Preparing Data in Data Warehousing

Abstract

A data warehouse (DW) stores data from various data sources to efficiently implement decision support or OLAP queries. Selecting data to load into a DW is one of the most important decisions in designing a DW. Most of the DW research activities in the last few years has been to select data in data sources that would provide the best performance for a given query pattern or workload. However, it is difficult to predict a query pattern or a workload due to the dynamic nature of decision support analysis. In this article we describe a component-based architecture for determining and prparing the source data to load into a DW. The architecture consists of 12 components that extract metadata from data sources, capture business rules, cleanse and profile source data, and map the source schemas to business rules. A methodology is presented that utilizes the components in correct order to select data in data sources to load into a DW. Third party tools used to carry out the tasks of some of the components are also dicussed. Decission support systems (DSS) are a key to gaining a competitive advantage for businesses. A data warehouse (DW) is a repository of integrated information that stores data from various data sources to efficiently implement decision support or OLAP queries. One of the most important decisions in designing a DW is to select an appropriate set of data in data sources to load into the DW. To solve this "source data selection" problem, substantial efforts have been made in the research community in the last few years to select an appropriate set of data that would provide the best performance for a given query pattern.