Applications of Multimodal Signal Processing to Combat COVID-19 Misinformation and Contact Tracing

In this talk, I will discuss two projects related to applications of multimodal signal processing to combat Covid 19. I begin with the problem of detecting misinformation in multimodal social media posts related to Covid-19. We collect one million Tweeter posts containing both video and text over a one year period, develop representation learning methods and contrastive learning and masked langugae modeling architectures for detecting inconsistencies between the two modalities, and characterize the performance of the proposed methods. Our best performaing method outperforms state of the art methods by 9 to 14% in accuracy. Next, I will describe machine learning methods desgined to detect proixmity of users as it relates to exposure notification and contact tracing in the context of Covid 19 transmission. We estimate proximity of any two users in indoor environments by applying machine learning techniquese to the temporal trace of the magnetometer and WiFi signal strength of their mobile devices to access points. We characterize the performance of both systems inside multiple buildings with hetereogeneous set of mobile devices. We find the WiFi and magnetometer proximity detection to achieve balanced accuracies of up to 78% and 91% respectively.