TENSOR COMPLETION FOR EFFICIENT AND ACCURATE HYPERPARAMETER OPTIMISATION IN LARGE-SCALE STATISTICAL LEARNING

Hyperparameter optimisation is a prerequisite for state-of-the-art performance in machine learning, with current strategies including Bayesian optimisation, hyperband, and evolutionary methods. While such methods improve performance, none of these is designed to explicitly take advantage of the underlying data structure. To this end, we introduce a completely different approach based on low-rank tensor completion. This is achieved by first forming a multi-dimensional tensor which comprises performance scores for different combinations of hyperparameters. Based on the realistic assumption that the so-formed tensor has a low-rank structure, reliable estimates of the unobserved validation scores of different combinations are obtained through tensor completion, from only a fraction of the known elements in the tensor. Through extensive experimentation, the proposed method is shown to exhibit competitive or superior performance to state-of-the-art hyperparameter optimisation strategies.