UI distribution technology based on ai and accessibility service for multi-device interaction

Abstract

With the increasing demand for multi-device interactions, numerous technologies such as Chromecast, Android auto, and Sidecar are being developed to support this trend. Existing multi-device technologies often incur development costs due to app modifications, are restricted to specific platforms, or require operating system modifications. Moreover, in cases of UI-level distribution, there are cases where only pre-defined UIs specified by developers can be distributed or only Android native UIs can be distributed. We propose a novel UI distribution system aimed at overcoming these limitations. This system leverages the Android accessibility service and a trained Faster R-CNN model for UI detection and distribution. Without the need for OS modifications, the system successfully detects and distributes various types of UIs from legacy apps, enabling interaction across different devices. The effectiveness of the proposed system has been validated through successful applications on 20 apps available on the Google Play Store.