Splines and Imaging: From Compressed Sensing to Neural Nets

The intent of this webinar is to demonstrate the optimality of splines for the resolution of inverse problems in imaging and the design of deep neural networks. To that end, I first present a representer theorem that states that the extremal points of a broad class of linear inverse problems with a generalized total-variation constraint are adaptive splines whose type is linked to the underlying regularization operator. When the latter is the second-order derivative operator, then the optimal reconstruction is an adaptive piecewise-linear spline with the smallest possible number of breakpoints. These splines are intrinsically sparse, and hence compatible with the kind of formulation (and algorithms) used in compressed sensing. I then show that these results are relevant to the investigation of neural networks as well. In particular, they yield a functional interpretation of shallow, infinite-width ReLU neural nets. Sparse adaptive splines also turn out to be ideally suited for the specification of deep neural networks with free-form activations.