Journal
INTERNATIONAL JOURNAL OF DATA SCIENCE AND ANALYTICS
Volume 6, Issue 4, Pages 327-339Publisher
SPRINGERNATURE
DOI: 10.1007/s41060-018-0119-9
Keywords
Bayesian optimization; Gaussian process; Stable Bayesian optimization; Acquisition function
Funding
- Australian Government through the Australian Research Council (ARC)
- Telstra-Deakin Centre of Excellence in Big Data and Machine Learning
- ARC Australian Laureate Fellowship [FL170100006]
- Australian Research Council [FL170100006] Funding Source: Australian Research Council
Ask authors/readers for more resources
Tuning hyperparameters of machine learning models is important for their performance. Bayesian optimization has recently emerged as a de-facto method for this task. The hyperparameter tuning is usually performed by looking at model performance on a validation set. Bayesian optimization is used to find the hyperparameter set corresponding to the best model performance. However, in many cases, the function representing the model performance on the validation set contains several spurious sharp peaks due to limited datapoints. The Bayesian optimization, in such cases, has a tendency to converge to sharp peaks instead of other more stable peaks. When a model trained using these hyperparameters is deployed in the real world, its performance suffers dramatically. We address this problem through a novel stable Bayesian optimization framework. We construct two new acquisition functions that help Bayesian optimization to avoid the convergence to the sharp peaks. We conduct a theoretical analysis and guarantee that Bayesian optimization using the proposed acquisition functions prefers stable peaks over unstable ones. Experiments with synthetic function optimization and hyperparameter tuning for support vector machines show the effectiveness of our proposed framework.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available