Atmospheric circulation regimes: Can cluster analysis provide the number?

The existence of multiple regimes in the extratropical tropospheric circulation is a hypothesis of theoretical importance with potential practical consequences. It is also a controversial hypothesis, and an abundance of conflicting results regarding both the existence and the number of regimes can be found in the literature. Studies of atmospheric regime behavior are often based on clustering methods such as k-means and mixture models. In the basic implementation of these methods the number of clusters has to be specified a priori and How many clusters? is a highly nontrivial question. For the mixture model a procedure to assess the number of clusters by cross validation has recently been introduced. For the k-means model a Monte Carlo test is introduced that compares the clustering of the original data with the clustering of Gaussian distributed surrogate data. The robustness of these methods and their ability to produce the right number of clusters is critically assessed. The study is based on both idealized data and atmospheric data. It is shown that applying the clustering methods to the Northern Hemisphere winter tropospheric geopotential heights gives conflicting and fragile results. In particular the number of clusters depends both on the clustering algorithm and on the period considered. Furthermore, the clustering methods find multiple clusters when applied to data similar to the atmospheric data but drawn from a unimodal, skewed distribution. It is also shown that both clustering methods report multiple clusters for idealized data drawn from distributions that are skewed or platykurtic but otherwise smooth and without bumps or shoulders. In these cases the number of clusters found depends on the sample size. In particular, for the mixture model the number of clusters increases without bounds with increasing sample size. It is concluded that in the atmospheric dataset studied the clustering methods provide only weak evidence for multiple regimes although the data is non-Gaussian with high statistical significance. It is also concluded that statistical models with basically unknown properties should be approached with utmost care or avoided completely.