BadRes: Reveal the Backdoors through Residual Connection

Generally, residual connections are indispensable network components in building Convolutional Neural Networks(CNNs) and Transformers for various downstream tasks in Computer Vision(CV), which encourages skip/short cuts between network blocks. However, the layer-by-layer loopback residual connections may also hurt the model's robustness by allowing unsuspecting input. In this paper, we proposed a simple yet strong backdoor attack method called BadRes, where the residual connections play as a turnstile to be deterministic on clean inputs while unpredictable on poisoned ones. We have performed empirical evaluations on four datasets with ViT and BEiT models, and the BadRes achieves 97% attack success rate without any performance degradation on clean data. Moreover, we analyze BadRes with state-of-the-art defense methods and reveal the fundamental weakness lying in residual connections.