Audio-Visual Event Recognition Through The Lens Of Adversary

As audio/visual classification models are widely deployed for sensitive tasks like content filtering at scale, therefore, it is critical to understand their robustness. This work aims to study several key issues related to multimodal learning through the lens of adversarial noises: 1) The trade-off between early/late fusion in terms of robustness 2) How does different frequency/time domain feature contribute to the robustness? 3) How does different neural modules contribute against the adversarial noise? In our experiment, we construct adversarial examples to attack state-of-the-art neural models trained on Google AudioSet and analyzed how much attack potency in terms of $\epsilon$ using different $L_p$ norms we would need to ``deactivate" the victim model. Using adversarial noise to dissect multi-modal models, we are able to provide an insight into what could be the best fusion strategy to balance the model parameters/accuracy and robustness trade-off and distinguish the robust features versus the non-robust features that various neural networks tend to learn.