Skip to main content

Semi-Supervised Multimodal Image Translation For Missing Modality Imputation

Wangbin Sun, Fei Ma, Yang Li, Shao-Lun Huang, Shiguang Ni, Lin Zhang

  • SPS
    Members: Free
    IEEE Members: $11.00
    Non-members: $15.00
    Length: 00:13:56
11 Jun 2021

Missing data is a common problem in multimodal and multi-view learning. It raises a critical challenge for most multimodal algorithms, which are unable to deal with incomplete datasets. Rather than discarding entries with missing modalities, this paper aims to reconstruct the complete image-based multimodal data by imputing missing modalities. We solve the imputation problem as an image translation task, which transforms images in one domain to other domains. Existing image translation techniques either can not fully utilize the information contained in partially complete entries or are limited to the bimodal situation. We propose a semi-supervised algorithm for multimodal learning with missing data, namely Cyclic Autoencoder (CycAE). Specifically, a novel cyclical structure, as well as the correlation among modalities, is integrated to leverage information from complete entries to incomplete ones. Experiments on two multimodal datasets show that our model outperforms state-of-the-art models. Downstream tasks can also benefit from the completed datasets.

Chairs:
Chaker Larabi

Value-Added Bundle(s) Including this Product

More Like This