SPARSE CNN ARCHITECTURE SEARCH (SCAS)

Advent of deep neural networks has revolutionized Computer Vision. However, designing of such models with high accuracy and low computation requirements is a difficult task and needs extensive human expertise. Recent advances in Neural Architecture Search use various methods like Deep Reinforcement Learning, Evolutionary methods, Gradient Descent, Hyper-Networks etc. to automatically generate neural networks with high level of accuracy. However, large size of such generated models limit their practical use. Recent findings about lottery ticket hypothesis suggest the existence of sparse subnetworks (winning tickets) which can reach the accuracy comparable to that of original dense network. In this paper, we present a method for leveraging redundancies inherent to deep Convolutional
Neural Networks (CNN) to guide the generation of sparse CNN models (to find the architectures with winning tickets) without significant loss in accuracy. We evaluate our proposed method with different NAS methods on CIFAR-10, CIFAR-100 and MNIST datasets. Our results show a reduction ranging from 2X to 12X in terms of model size and 2X to 19X in terms of number of MAC operations with less than 1% drop in accuracy.