Improvement in precision of counting actinic keratoses

Background Actinic keratoses (AKs) often serve as a primary endpoint for clinical studies. However, reliability of counting these lesions is poor, even among expert dermatologists. Objectives To investigate the reliability of counting AKs before and after a yearly consensus meeting, held annually for 4 years. Methods As part of the Veterans Affairs (VA) Keratinocyte Carcinoma Chemoprevention Trial, board-certified dermatologists convened annually for 4 years to individually count the number of actinic keratoses on three to five test subjects. The dermatologists then met as a group for a consensus discussion on what constituted an AK lesion on each subject. Afterwards, each dermatologist repeated the independent counting exercise on three to five new subjects. The intraclass correlation coefficient (ICC) was used to analyze the reliability of counting AKs among the dermatologists. Results Eight dermatologists participated in this exercise for 4 consecutive years. Pre-consensus discussion ICCs over 4 years were 0.18, 0.34, 0.38, 0.75, respectively, showing sustained improvement with each consensus discussion. The greatest improvement in reliability of AK counts was shown during the first year of consensus discussions, when the ICC improved from 0.18 to 0.67. There was no improvement by the fourth year of consensus discussion, with pre- and post-consensus ICCs of 0.75 and 0.75, respectively. Conclusions Annual consensus discussions can lead to improvement in reliability of AK counts. This improvement was sustained over 4 years. By the fourth year, the discussion meeting had no effect on improvement in reliability. A consensus meeting discussion may be helpful for improving reliability in other trials.