Efficient two-stage inference for sign language in real-time

Abstract

Hearing impaired people use sign language to communicate with other people. Unfortunately, average people do not know about sign language, so hearing impaired people are experiencing a difficulty in government offices or banks because they are hard to communicate with officers. To address this problem, cloud services or studies for interpreting sign language through artificial intelligence are emerging recently . However, among them, the Korean sign language interpretation model places substantial computation burden on the cloud server due to heavy deep learning model and unnecessary computations. In this paper, we propose a two-stage inference system that greatly reduces the burden on cloud servers by effective task division. In particular, we focus on the front-end kiosk, which only serves to store and send videos in the existing Interpreting system. After deploying a deep learning model that is trained to detect sign language in the front-end kiosk, by sending only sign language videos instead of sending videos always to the cloud server, we can significantly reduce the workloads of the cloud server. In addition, through optimizations of the system, the amount of computation in the sign language detection model can be reduced by more than six times. In this way, we minimize additional delays in the kiosk by adapting the sign language detection model to the constraints of the kiosk's computational resources. Finally, by using frame interpolation and call sign method, and improving sign language detection process, we can achieve a better system accuracy in corner cases as well as in general cases.