Optimization-Based Neural Networks Compression

This paper presents a method for constructing a \textit{size compressed} neural network with better or similar accuracy than a given dense neural network, therefore the compressed network requires less memory and computational resources. The presented method relies basically on learning successive mappings between the given dense neural network (teacher) hidden features and the size-compressed neural network (student) hidden features, where the latter is learned also to solve the initial task of the teacher network. The presented method is particularly compared to baselines where we specifically show that the additional learned mappings significantly improve the performance (accuracy and computation) of the student network.