Distributed Deep Variational Information Bottleneck

This paper deals with a distributed version of the information bottleneck method. For this problem, we develop a variational bound on the optimal tradeoff between relevance and complexity that generalizes the evidence lower bound (ELBO) to the distributed setting. Furthermore, we also provide a variational inference type algorithm that allows to compute this bound and in which the mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Experimental results are provided to support the efficiency of the approaches and algorithms which we develop in this paper.