Energy-efficient object recognition algorithm and hardware architecture for mobile vision platform

Abstract

In recent years, object recognition has been widely adopted in various real-life applications. Mi-crosoft’s Kinect uses body-part recognition as a gaming interface, and automakers such as Toyota and BMW incorporate vehicle, pedestrian, and lane detection in their advanced driver-assistance systems. Smartphones that operate within a low power budget also use object recognition for booming applications such as augmented reality, face-recognition-based security, and gesture-recognition-based user interfaces. In such applications, the Scale Invariant Feature Transform (SIFT) is the most popular candidate for how to extract some interest points out of the objects and describe them in a way that invariants to translation, scal-ing, and rotation. However, SIFT-based object recognition consumes a lot of power because of the heavy computations required in descriptor generation and matching. In addition, today’s high-resolution image sensors and tight power budgets make real-time SIFT implementation in mobile devices even harder

recent mobile cameras provide more than 720p resolution at 30 frames per second (fps), while the power consump-tion on mobile CPUs and GPUs ranges from roughly 0.1 W to 1 W. To realize real-time SIFT-based object recognition that meets these high resolution and low power requirements, we propose an object-recognition processor using (1) an attention-based recognition algorithm for energy efficiency, (2) a heterogeneous multicore architecture for data and thread parallelism, and (3) per-formance model based fine-grain dynamic voltage and frequency scaling. The processor determines regions of interest (ROIs)─the parts of the image that likely contain target objects─which lets us perform the main recognition on only the selected regions, minimizing unnecessary computations. The heterogeneous multicore architecture provides several types of parallelism and so achieves high throughput and low power consump-tion for highly parallelizable recognition processing. The high-bandwidth NoC plays a role as the communica-tions backbone for tens of processing cores while meeting the high-resolution video sequence’s streaming de-mand of more than a few hundred megabytes per frame. To increase energy efficiency of the multi-core pro-cessor, the performance model based fine-grain dynamic voltage and frequency scaling is proposed for pro-cessing cores of 5-stage task-level pipeline. As a result, the fabricated SoC achieves 30 fps dynamic object recognition for unmanned aerial vehicle (UAV) system with 720p video streams while dissipating 320mW, achieving 2.54 times higher energy efficiency with 10.5nJ/pixel compared to the state-of-the-art vision pro-cessors.