CE-BART: Cause-and-Effect BART for Visual Commonsense Generation

Cited 1 time in webofscience Cited 0 time in scopus
  • Hit : 383
  • Download : 35
“A Picture is worth a thousand words”. Given an image, humans are able to deduce various cause-and-effect captions of past, current, and future events beyond the image. The task of visual commonsense generation has the aim of generating three cause-and-effect captions for a given image: (1) what needed to happen before, (2) what is the current intent, and (3) what will happen after. However, this task is challenging for machines, owing to two limitations: existing approaches (1) directly utilize conventional vision–language transformers to learn relationships between input modalities and (2) ignore relations among target cause-and-effect captions, but consider each caption independently. Herein, we propose Cause-and-Effect BART (CE-BART), which is based on (1) a structured graph reasoner that captures intra- and inter-modality relationships among visual and textual representations and (2) a cause-and-effect generator that generates cause-and-effect captions by considering the causal relations among inferences. We demonstrate the validity of CE-BART on the VisualCOMET and AVSD benchmarks. CE-BART achieved SOTA performance on both benchmarks, while an extensive ablation study and qualitative analysis demonstrated the performance gain and improved interpretability. © 2022 by the authors.
Publisher
MDPI
Issue Date
2022-12
Language
English
Article Type
Article
Citation

SENSORS, v.22, no.23

ISSN
1424-8220
DOI
10.3390/s22239399
URI
http://hdl.handle.net/10203/303483
Appears in Collection
EE-Journal Papers(저널논문)
Files in This Item
127443.pdf(1.68 MB)Download
This item is cited by other documents in WoS
⊙ Detail Information in WoSⓡ Click to see webofscience_button
⊙ Cited 1 items in WoS Click to see citing articles in records_button

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0