Automatic identification and back-transliteration of foreign words for information retrieval

Abstract

Many foreign words and English words appear in Korean texts, especially in the areas of science and engineering. We recognize two issues related to foreign words, which should be addressed for information retrieval (IR). First, since foreign words are introduced dynamically and not always found in a dictionary, they cause problems in morphological analysis required for indexing. Second, although a foreign word and its origin in the source language like English refer to the same concept, they are erroneously treated as independent index terms. As a way of alleviating the first problem we developed an algorithm that first identifies a phrase containing a foreign word and then extracts the foreign word part from the phrase based on statistical information. For the second problem, we present our method for back-transliteration of a foreign word to its English origin. Finally we report our evaluation results for each of the algorithms;md experimental results for their impact on IR effectiveness. (C) 1999 Elsevier Science Ltd. All rights reserved.