In this letter, we propose the photo-responsible synaptic devices by using stackable GaAs photodetectors (PDs) and Ge-on-insulator (Ge-OI) synaptic transistors for the future three-dimensional (3D) artificial vision sensors. The photo-responsible synapse showed good photo-responding synaptic behaviors depending on the incident light to GaAs PD, which changes the hole injection into the Ge-OI transistors, resulting in the change in potentiation/depression characteristics. The training simulation highlighted the fabricated photo-responsible synapse can provide high colored pattern recognition tasks for semiconductor-based hardware systems.