Fitting limit lines (envelope curves) to spreads of geoenvironmental data

Geoscientists frequently are interested in defining the overall trend in x-y data clouds using techniques such as least-squares regression. Yet often the sample data exhibits considerable spread of y-values for given x-values, which is itself of interest. In some cases, the data may exhibit a distinct visual upper (or lower) 'limit' to a broad spread of y-values for a given x-value, defined by a marked reduction in concentration of y-values. As a function of x-value, the locus of this 'limit' defines a 'limit line', with no (or few) points lying above (or below) it. Despite numerous examples of such situations in geoscience, there has been little consideration within the general geoenvironmental literature of methods used to define limit lines (sometimes termed 'envelope curves' when they enclose all data of interest). In this work, methods to fit limit lines are reviewed. Many commonly applied methods are ad-hoc and statistically not well founded, often because the data sample available is small and noisy. Other methods are considered which correspond to specific statistical models offering more objective and reproducible estimation. The strengths and weaknesses of methods are considered by application to real geoscience data sets. Wider adoption of statistical models would enhance confidence in the utility of fitted limits and promote statistical developments in limit fitting methodologies which are likely to be transformative in the interpretation of limits. Supplements, a spreadsheet and references to software are provided for ready application by geoscientists.

Automated summary

Geoscientists often use techniques like least-squares regression to define trends in x-y data clouds, but sometimes the data exhibits a wide spread of y-values for given x-values, with visual upper or lower limits. The paper reviews methods for fitting limit lines, concluding that commonly used ad-hoc methods may lack statistical robustness, while other methods corresponding to specific statistical models offer more objective estimation. The adoption of statistical models could enhance confidence in fitted limits and promote transformative developments in limit fitting methodologies for interpretation of limits.