Improved Covariance Matrix Estimation With An Application In Portfolio Optimization

One of the major challenges in multivariate analysis is the estimation of population covariance matrix from the sample covariance matrix (SCM). Most recent covariance matrix estimators use either shrinkage transformations or asymptotic results from Random Matrix Theory (RMT). Both of these techniques try to achieve a similar goal which is to remove noisy correlations and add structure to SCM to overcome the bias-variance trade-off. Both methods have their respective pros and cons. In this paper, we propose an improved estimator which exploits the advantages of these techniques by taking optimally weighted convex combination of covariance matrices estimated by shrinkage transformation and a filter based on RMT. It is a generalized estimator which can adapt to changing sampling noise conditions by performing hyperparameter optimization. Using data from six of the world's biggest stock exchanges, we show that the proposed estimator outperforms the existing estimators in minimizing the out-of-sample risk of the portfolio and hence predicts population statistics more precisely. The proposed estimator can be useful in a wide range of machine learning and signal processing applications.