Inter-rater agreement between humans and computer in quantitative assessment of computed tomography after cardiac arrest

BackgroundHead computed tomography (CT) is used to predict neurological outcome after cardiac arrest (CA). The current reference standard includes quantitative image analysis by a neuroradiologist to determine the Gray-White-Matter Ratio (GWR) which is calculated via the manual measurement of radiodensity in different brain regions. Recently, automated analysis methods have been introduced. There is limited data on the Inter-rater agreement of both methods. MethodsThree blinded human raters (neuroradiologist, neurologist, student) with different levels of clinical experience retrospectively assessed the Gray-White-Matter Ratio (GWR) in head CTs of 95 CA patients. GWR was also quantified by a recently published computer algorithm that uses coregistration with standardized brain spaces to identify regions of interest (ROIs). We calculated intraclass correlation (ICC) for inter-rater agreement between human and computer raters as well as area under the curve (AUC) and sensitivity/specificity for poor outcome prognostication. ResultsInter-rater agreement on GWR was very good (ICC 0.82-0.84) between all three human raters across different levels of expertise and between the computer algorithm and neuroradiologist (ICC 0.83; 95% CI 0.78-0.88). Despite high overall agreement, we observed considerable, clinically relevant deviations of GWR measurements (up to 0.24) in individual patients. In our cohort, at a GWR threshold of 1.10, this did not lead to any false poor neurological outcome prediction. ConclusionHuman and computer raters demonstrated high overall agreement in GWR determination in head CTs after CA. The clinically relevant deviations of GWR measurement in individual patients underscore the necessity of additional qualitative evaluation and integration of head CT findings into a multimodal approach to prognostication of neurological outcome after CA.

Automated summary

Background computer tomography (CT) is used to predict neurological outcome after cardiac arrest (CA). This study compared the inter-rater agreement between human and computer evaluators in determining the Gray-White-Matter Ratio (GWR) and observed deviations in individual patients. The results showed that although there were some deviations, they did not lead to false prediction of poor neurological outcomes at the given threshold.