Abstract

We present a new methodology for assessing when data-based predictive models can be trusted. Particularly, we propose to learn a model from experimentation that determines, for a given labeled data set and a learning technique, when the model generated by the respective technique on the given data can be trusted to perform within specified accuracy limits. That is to say, we apply machine learning to machine learning: We repeatedly use a technique to generate models, referred as primary model, for a supervised regression problem. Based on the resulting model performance on a hold-out validation set, we then learn when the trained primary model can be expected to perform well and when there is a concern regarding the trustworthiness of that model.