The Bethesda interobserver reproducibility study (BIRST) - A web-based assessment of the Bethesda 2001 System for classifying cervical cytology

BACKGROUND. The Bethesda System (TBS) along with its companion atlas was updated in 2001 to improve standardization, clarity, and reproducibility of cervical cytology reporting. METHODS. The authors used a novel web-based format to compare assessments of 77 images demonstrating a range of classical and borderline cytologic changes by a self-selected group of United States cytotechnologists (n = 216) and pathologists (n = 185). RESULTS. Participants were highly experienced, with 71.2% of cytotechnologists and 53.0% of pathologists reporting > 10 years of practice. The mean percentage of exact agreement with the panel was slightly though significantly higher for cytotechnologists (57.0%) compared with pathologists (53.4%), adjusted for experience (P=.004); cervical cytology percentage effort (P=.0005); or cervical accession volume (P=.0002). Compared with the TBS panel, exact agreement was achieved for 55.1% of image ratings compared with 82.3% agreement at the level of Negative vs non-Negative for images with a single-panel interpretation. Agreement with the panel was highest for images classified as Low-Grade Squamous Intraepithelial Lesion and lowest for Atypical Squamous Cells qualified as either of Undetermined Significance or Cannot Exclude a High-Grade Squamous Intraepithelial Lesion. Reviewers were less sensitive in identifying high-grade glandular lesions than they were in identifying high-grade squamous lesions at any threshold (P <.001). CONCLUSIONS. Morphologic appearances of images were more important determinants than participants' academic or professional degrees with regard to inter-observer reproducibility in classifying cervical cytology images. Experienced cytotechnologists and pathologists performed similarly Participants achieved higher sensitivity for identifying high-grade squamous lesions than they did for high-grade glandular lesions. These findings demonstrated that web-based studies may be useful in assessing interobserver agreement in classifying images.