Air-Text: Air-Writing and Recognition System

Abstract

Text entry takes an important role of effectively delivering the intention of users to computers, where physical and soft keyboards have been widely used. However, with the recent trends of developing technologies like augmented reality and increasing contactless services due to COVID-19, a more advanced type of text entry is required. To tackle this issue, we propose Air-Text which is an intuitive system to write in the air using fingertips as a pen. Unlike previously suggested air-writing systems, Air-Text provides various functionalities by the seamless integration of air-writing and text-recognition modules. Specifically, the air-writing module takes a sequence of RGB images as input and tracks both the location of fingertips (5.33 pixel error in 640x480 image) and current hand gesture class (98.29% classification accuracy) frame by frame. Users can easily perform writing operations such as writing or deleting a text by changing hand gestures, and tracked fingertip locations can be stored as a binary image. Then the text-recognition module, which is compatible with any pre-trained recognition models, predicts a written text in the binary image. In this paper, examples of single digit recognition with MNIST classifier (96.0% accuracy) and word-level recognition with text recognition model (79.36% character recognition rate) are provided.