EXPLORING THE EFFECT OF L0/L2 REGULARIZATION IN NEURAL NETWORK PRUNING USING THE LC TOOLKIT

The LC Toolkit is an open-source library written in Python and PyTorch that allows to compress any neural network using several compressions including quantization, pruning, and low-rank. The versatility of the framework is rooted in the principled mathematical formulation of the underlying network compression problems with subsequent optimization by learning-compression (LC) algorithm. In this paper, we utilize the LC toolkit's common algorithmic base to take a deeper look into $\ell_0$-constrained pruning problems defined as follows: given a budget of $\kappa$ non-zero weights, which weights should be pruned in the final network? We observe that $\ell_0$-pruned networks have a different connectivity structure compared to pruning results using $\ell_1$ norm. We propose a change to the formulation of the problem involving a small amount of $\ell_2$ weight decay which has a favorable effect on connectivity structure. We study the properties of the proposed $\ell_0 + \ell_2$ formulation using the LC toolkit and empirically demonstrate that such a scheme achieves a competitive sparsity-error tradeoff while having better structural sparsity.