Rethinking Training Schedules For Verifiably Robust Networks

New and stronger adversarial attacks can threaten existing defenses. This possibility highlights the importance of certified defense methods that train deep neural networks with verifiably robust guarantees. A range of certified defense methods has been proposed to train neural networks with verifiably robustness guarantees, among which Interval Bound Propagation (IBP) and CROWN-IBP have been demonstrated to be the most effective. However, we observe that CROWN-IBP and IBP are suffering from Low Epsilon Overfitting (LEO), a problem arising from their training schedule that increases the input perturbation bound. We show that LEO can yield poor results even for a simple linear classifier. We also investigate the evidence of LEO from experiments under conditions of worsening LEO. Based on these observations, we propose a new training strategy, BatchMix, which mixes various input perturbation bounds in a mini-batch to alleviate the LEO problem. Experimental results on MNIST and CIFAR-10 datasets show that BatchMix can make the performance of IBP and CROWN-IBP better by mitigating LEO.