Emergence of multimodal action representations from neural network self-organization

Cited 5 time in webofscience Cited 0 time in scopus
  • Hit : 214
  • Download : 144
The integration of multisensory information plays a crucial role in autonomous robotics to forming robust and meaningful representations of the environment. In this work, we investigate how robust multimodal representations can naturally develop in a self-organizing manner from co-occurring multisensory inputs. We propose a hierarchical architecture with growing self-organizing neural networks for learning human actions from audiovisual inputs. The hierarchical processing of visual inputs allows to obtain progressively specialized neurons encoding latent spatiotemporal dynamics of the input, consistent with neurophysiological evidence for increasingly large temporal receptive windows in the human cortex. Associative links to bind unimodal representations are incrementally learned by a semi-supervised algorithm with bidirectional connectivity. Multimodal representations of actions are obtained using the co-activation of action features from video sequences and labels from automatic speech recognition. Experimental results on a dataset of 10 full-body actions show that our system achieves state-of-the-art classification performance without requiring the manual segmentation of training samples, and that congruent visual representations can be retrieved from recognized speech in the absence of visual stimuli. Together, these results show that our hierarchical neural architecture accounts for the development of robust multimodal representations from dynamic audiovisual inputs. (C) 2016 The Authors. Published by Elsevier B.V.
Publisher
ELSEVIER SCIENCE BV
Issue Date
2017-06
Language
English
Article Type
Article
Keywords

SUPERIOR TEMPORAL SULCUS; MULTISENSORY INTEGRATION; AUDIOVISUAL INTEGRATION; INVERSE EFFECTIVENESS; ADULT NEUROGENESIS; RECEPTIVE WINDOWS; BIOLOGICAL MOTION; HUMAN BRAIN; PERCEPTION; CORTEX

Citation

COGNITIVE SYSTEMS RESEARCH, v.43, pp.208 - 221

ISSN
1389-0417
DOI
10.1016/j.cogsys.2016.08.002
URI
http://hdl.handle.net/10203/224711
Appears in Collection
EE-Journal Papers(저널논문)
Files in This Item
000402461200018.pdf(1.24 MB)Download
This item is cited by other documents in WoS
⊙ Detail Information in WoSⓡ Click to see webofscience_button
⊙ Cited 5 items in WoS Click to see citing articles in records_button

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0