Incorporating interpreter variability into estimation of the total variance of land cover area estimates under simple random sampling

Area estimates of land cover and land cover change are often based on reference class labels determined by analysts interpreting satellite imagery and aerial photography. Different interpreters may assign different reference class labels to the same sample unit. This interpreter variability is typically not accounted for in variance estimators applied to area estimates of land cover. A simple measurement model provides the basis for an estimator of the total variance (V-Total) that takes into account both sampling variance and interpreter variance. This method requires two or more reference class interpretations (i.e., repeated measurements) obtained by analysts, working independently of each other, for the full sample or a random subsample of the full sample. Estimators of the total variance ((V) over cap (Total)) and the variance component attributable to interpreters ((V) over cap (1)) were obtained for the case of two reference class interpretations per repeated sample unit. To evaluate the effect of interpreter variability on variance estimation, we used land cover reference data interpreted by seven analysts who each interpreted the same 300 sample pixels from a region of the Pacific Northwest of the United States. From these data, we estimated the contribution of interpreter variance to the total variance (i.e., (V) over cap/(V) over cap (Total)) and the relative bias of the standard simple random sampling variance estimator ((V) over cap (stand)) as an estimator of V-Total, defined as 100%*((V) over cap (stand) - (V) over cap (Total))/(V) over cap (Total). For each of five land cover classes, we computed (V) over cap (1), (V) over cap (Total), and (V) over cap (stand) using the sample data from each of the 21 possible pairwise combinations of the seven interpreters, and then calculated the mean of (V) over cap (1)/(V) over cap (Total) and the mean of the estimated relative bias of (V) over cap (stand) over these 21 pairs. Based on the mean of (V) over cap (1)/(V) over cap (Total) per class, interpreter variance contributed from 25% (cropland) to 76% (grass/shrub) of the total variance, indicating that interpreter variance was a non-negligible component of the total variance. Typically, the standard variance estimator, (V) over cap (stand), underestimated the total variance with the mean estimated relative bias ranging from 3% (cropland) to 33% (grass/shrub). Classes with greater inconsistency between pairs of interpreters had larger contributions of interpreter variance to the total variance ((V) over cap (1)/(V) over cap (Total)) and larger negative estimated relative bias of (V) over cap (stand). Given that interpreter variance can contribute substantially to the total variance, the repeated measurements approach offers a practical way to incorporate this variability into an estimator of the total variance.

Automated summary

Area estimates of land cover and land cover change are often determined by analysts interpreting satellite imagery and aerial photography. However, there is variability in the reference class labels assigned by different interpreters to the same sample units, which is not usually considered in variance estimators. By using a simple measurement model and repeated measurements by independent analysts, it was found that interpreter variance can significantly contribute to the total variance, and the standard variance estimator may underestimate the total variance due to this variability. The repeated measurements approach offers a practical way to incorporate interpreter variability into the total variance estimator, highlighting the importance of accounting for interpreter variance in land cover area estimation.