Clustering-based feature detection and tracking for event cameras

Abstract

as a collection of events called event streams, which renders most computer vision algorithms ineffective. Despite the camera's unique data output, many event feature detection and tracking algorithms have shown significant progress by making detours to frame-based data representations. This paper questions the need to do so, and proposes a solution using the raw events for feature detection and tracking. Our methods, event Clustering-based Detection and Tracking (eCDT) and its variant (eCDT$^+$), cluster adjacent polarity events to retrieve event trajectories. By noticing the geometric relationship that opposing polarity events act as barriers separating unique feature clusters, we devise a clustering approach that can robustly obtain equal polarity event tracks. The clustering approach called k-NN Classifier-based Spatial Clustering and Applications with Noise (KCSCAN) takes inspiration from density-based spatial clustering of applications with noise (DBSCAN) and k-nearest neighbor (k-NN) classifier. Therefore, eCDT, contrary to other feature detection and tracking, utilizes an approach that is original to event data. We additionally propose a descriptor for event clusters that are used in eCDT$^+$ to find reappearing cluster tracks, elongating the feature tracking. Thanks to our clustering approach in spatio-temporal space, our method automatically solves the problem of finding good features to track or finding good trackers for a detector, by performing two tasks with one method. Also, eCDT can extract feature tracks at any frequency with an adjustable time window. Our method achieves better feature tracking ages compared to the state-of-the-art approaches while having a low error approximately equal to those of state-of-the-art approaches. Therefore, we prove that raw events are enough to implement feature detection and tracking.

Event cameras are novel sensors with advantageous capabilities, leading to spurring interests in event camera research. However, these cameras interpret the world in an entirely different manner