The costs of complex model optimization

Each data-driven action in data modelling consumes degrees of freedom, whether it concerns estimation of parameters, estimation of meta-parameters or selecting variables. By using a double cross validation approach for degrees of freedom calculation the costs for meta-parameter estimation and variable selection can be determined explicitly. The only assumptions are independent and identically distributed errors, which make the approach applicable to many predictive modelling techniques. Published by Elsevier B.V.