Analysis of intra-observer and inter-observer variability of pathologists for non-benign thyroid fine needle aspiration cytology according to Bethesda system categories

Aim To examine the intra- and inter-observer variability for non-benign thyroid cytological subcategories according to the Bethesda classification system after the second review. Methods Between November 2018 and May 2019, thyroid fine needle aspiration biopsies of 381 nodules were retrospectively evaluated. Among them, 74 non-benign (category III-VI) thyroid biopsies, analyzed according to the Bethesda system (pathologist 1:40 vs pathologist 2:34) by two independent pathologists, were reassessed by the same pathologists and by a cytopathologist. In this observer-blinded study, weighted Cohen's kappa was used to assess the intra-observer agreement, and Krippendorff's alpha was used to assess the inter-observer agreement. Results At the first and second evaluations of pathologists 1 and 2, the percentage agreement was 62.5% for pathologist 1 and 58.8% for pathologist 2. The intra-observer agreement was substantial (kappa = 0.705) for pathologist 1, and moderate (kappa = 0.447) for pathologist 2. In the second evaluation of pathologist 1 and 2, which was compared with the cytopathologist, the agreement percentage of pathologist 1 with the cytopathologist was 50.0%, and that of pathologist 2 was 56.8%. The inter-observer agreement was below the lowest acceptable limit for an overall agreement (alpha = 0.634) among the three raters. The inter-observer agreement was only acceptable between the cytopathologist and the second pathologist, while it was low between the other raters. In the evaluation of the non-benign nodules, the mean category score of the cytopathologist was 3.22 and lower than both pathologists (3.73 and 3.58, respectively). Conclusions The intra-observer agreement of pathologists was moderate-to-substantial in the evaluation of non-benign thyroid biopsies according to the Bethesda reporting system. However, the inter-observer agreement was below the lowest acceptable limit when the cytopathologist was taken as a reference.

Automated summary

Moderate-to-substantial intra-observer agreement was found among pathologists in the evaluation of non-benign thyroid biopsies using the Bethesda reporting system. However, the inter-observer agreement fell below the lowest acceptable limit when the cytopathologist was considered as a reference.