Inter-observer variation in segmenting glioma on MRI before and after resection

Longitudinal MR images obtained for treating glioma patients contain valuable diagnostic and prognostic information. One way to extract this information is to delineate tumor related entities and compare these in large cohorts. Reproducible and reliable measurement of volume and location is a necessary condition to obtain valid results. Manual segmentation of glioma by an expert is considered to be the gold standard, and is often provided by a single rater. In this study the agreement between different raters is recorded in both enhancing and non-enhancing glioma before and after surgical treatment, and at progression (pre-op, post-op and progression respectively). A total of 40 patients were included, of which 20 have non-enhancing glioma, usually of WHO grade II or III, and 20 have enhancing glioma, usually glioblastoma. Inter-rater agreement is recorded with the generalized conformity index (Clgen). Three raters of the VUmc neurosurgery department have performed the segmentations. Patients with non-enhancing glioma show no enhancing elements with the exception of three patients in the progression time point and one patient in pre-op. The segmented non enhancing elements have a median Clgen of 0.64, 0.30, and 0.27 in the pre-op, post-op and progression respectively. For patients with enhancing glioma both enhancing and non enhancing elements are observed. The segmented enhancing elements have median Clgen scores of 0.86, 0.24, and 0.69 in pre-op, post-op and progression respectively. Median Clgen for the non-enhancing elements is 0.47, 0.07 and 0.20 in pre-op, post-op and progression respectively. A Jaccard-like score for object comparison of small objects is sensitive for small volumes, returning a low score for slight misalignment. When dilating small volume segmentations to 10 mL as compensation for small volume comparison with the Clgen the attained scores are not much improved indicating that different raters not only disagreed about object shape but also about location.