Diffusion Schrodinger bridge for unpaired day-to-night event translation

Abstract

Event camera with a high dynamic range ensures scene capture even in low-light conditions. However, events captured at night exhibit differences in patterns compared to events captured during the day, causing performance degradation when using event-based algorithms trained on only day events for night scenarios. To overcome the limitation, I first tackle day-to-night translation in the event modality within an unpaired setting. To this end, I propose a Schrödinger Bridge diffusion model that effectively learns the mapping function from one domain to another, tailored for day-to-night event translation. Throughout the translation, wavelet decomposition is applied to analyze shared and distinct features between day and night events, trying to preserve shared elements while transporting differences. The temporally disentangled encoder is also designed which explicitly separates the event features and the temporal dimension. Introducing temporal contrastive learning helps maintain temporal order in events during the translation process. To validate the efficacy of the proposed methodology, I redesign metrics for evaluating events translated in an unpaired setting, aligning them with the event modality for the first time and subsequently justifying the metrics. The results obtained from the metrics illustrate the superior performance of the proposed approach compared to alternative methods. Lastly, through event translation, I facilitate downstream networks trained by incorporating labeled day events alongside the translated synthetic night events. This approach effectively mitigates the performance degradation of applying real night events to the task.