(A) low-power gaze-activated object recognition system for smart glasses

Abstract

Smart glasses or head-mounted displays (HMDs) have been gaining traction as next-generation mainstream wearable devices. However, previous smart glasses have had limited application, primarily due to their lacking a smart user interface (UI) and user experience (UX). Since smart glasses have a small compact wearable platform, their UI requires new modalities, rather than a computer mouse or a 2D touch panel. Recent speech-recognition-based UIs require voice input to reveal the user’s intention to not only users of smart glasses but also others, which raises privacy concerns in a public space. In addition, prior works attempted to support object recognition (OR) or augmented reality (AR) in smart eyeglasses, but consumed considerable power, >381mW, resulting in <6 hours operation time with a 2100mWh battery. In this work, we propose a low-power OR system with an intention-concealed gaze UI for smart glasses, which can be used all day long with battery power. For the low-power gaze UI, we propose a low-power single-chip gaze estimation sensor, called Gaze Image Sensor (GIS). In GIS, a novel column-parallel pupil edge detection circuit (PEDC) with new pupil edge detection algorithm, XY-PD, is proposed which results in 2.9x power reduction with 16x larger resolution compared to previous work. Also, a logarithmic SIMD processor is proposed for robust pupil center estimation, <1 pixel error, with low-power floating-point implementation. For OR, low-power multi-core object recognition processor (ORP) is implemented. In ORP, task-level pipeline with keypoint-level scoring is proposed to reduce number of cores as well as operating frequency of keypoint matching processor (KMP) for low-power consumption. Also, dual-mode convolutional neural network processor (CNNP) is designed for fast tile selection without external memory accesses. In addition, a pipelined descriptor generation processor (DGP) with LUT-based non-linear operation is newly proposed for low-power OR. Last, DVFS for dynamic power reduction in ORP is applied. Combining both of the GIS and ORP fabricated in 65nm CMOS logic process, only 75mW average power consumption is achieved with real-time OR performance, which is 1.2x and 4.4x lower power than the previously published work.