Abstract

Bayesian deep learning is a method to quantity uncertainty for black-box models such as deep learning architectures. Despite the huge success of these models for several tasks such as computer vision and natural language processing, the existing performance metrics rely on point estimates and cannot properly explain the uncertainty around the predictions. Knowing the model confidence is absolutely necessary to avoid over-optimistic decisions. Therefore, in this paper we propose a novel F score that takes into account the Leave-One-Out (LOO) performance using posterior samples from stochastic gradient Markov chain Monte Carlo algorithms. The new Fιoo metric is evaluated for two different Bayesian models under flat and hierarchical priors. The results for the MNIST data set indicate that the new approach avoids errors from finite sample estimates.