The hazards of split-sample validation in hydrological model calibration

This paper investigates the issues related to the use of validation in hydrological model calibration. Traditionally, models are calibrated and then assessed on an independent period (split-sample) to determine their adequacy in simulating streamflow as compared to observations. In this study, two hydrological models and three North American catchments are used to evaluate the effects of using validation to assess the model parameters' robustness on the model's actual simulation capabilities and accuracy in simulating streamflow. The length of the calibration period is increased from 1 to 16 years, and for each case a large number of randomly selected combinations of years are used for calibration and for validation using the Nash-Sutcliffe Efficiency metric. The calibrated model is then run on an independent 8-year test-period to assess the model's actual performance in simulation mode in unknown conditions. The process is bootstrapped 30 times to ensure the robustness of the results. The tests pit the calibration/validation methods on increasing calibration period lengths against a full calibration on the entire available dataset. Results show that the calibration on the full dataset is the optimal strategy as it generates the most robust parameter sets, provides the best model accuracy on an independent testing period and does not require assumption-making on the modeler's part. The calibrated parameter sets for each test-case were evaluated using the relative bias and correlation metrics, which revealed that the method transfers well to these two other metrics. Results also demonstrate the pitfalls of the commonly used split-sampling strategy, where good parameter sets may be discarded due to model performance discrepancies between calibration and validation periods. The conclusions point to the need to use as many years as possible in the calibration step and to entirely disregard the validation aspect under certain conditions.