Stochastic modeling of Korean language for Open-vocabulary handwritten Hangul recognition

Abstract

The stochastic language model provides linguistic likelihoods of expressions that can be used in resolving ambiguities in geometric evidences. This can be especially beneficial in handwritten Hangul recognition, because Hangul characters have highly similar shapes due to the compositional nature. For modeling of Korean language, morpheme-based models have been preferred due to her agglutinative characteristics. These models basically assume that the input texts are syntactically well-formed. However, this assumption holds only in limited domains. Also, they demand morphologically analyzed corpus for training, which are expensive to use because the corpus needs to be processed by human experts. We present a novel language model that can be trained from a raw corpus. Without relying on the linguistic knowledge, we train the lexicon and their associated probabilities out of raw texts based on statistical measures. Experiments show that the proposed model effectively captures the variable-length regularities in Korean language even though no linguistic knowledge was utilized explicitly during training. In recognition experiments, both the character recognition rates and the word recognition rates show significant improvment by employing the proposed language model.