Journal
INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY
Volume 25, Issue 3-4, Pages 171-190Publisher
TAYLOR & FRANCIS INC
DOI: 10.1080/10508414.2015.1162636
Keywords
-
Categories
Funding
- German Federal Ministry for Economic Affairs and Energy via the Federal Aeronautical Research Program LuFo [IV-2 (20V0803)]
Ask authors/readers for more resources
Objective: The aim of this study was to analyze influences on interrater reliability and within-group agreement within a highly experienced rater group when assessing pilots' nontechnical skills. Background: Nontechnical skills of pilots are crucial for the conduct of safe flight operations. To train and assess these skills, reliable expert ratings are required. Literature shows to some degree that interrater reliability is influenced by factors related to the targets, scenarios, rating tools, or the raters themselves. Method: Thirty-seven type-rating examiners from a European airline assessed the performance of 4 flight crews based on video recordings using LOSA and adapted NOTECHS tools. We calculated r(wg) and ICC(3) to measure within-group agreement and interrater reliability. Results: The findings indicated that within-group agreement and interrater reliability were not always acceptable. It was shown that the performance of outstanding pilots was rated with the highest within-group agreement. For cognitive aspects of performance, interrater reliability was higher than for social aspects of performance. Agreement was lower on the pass-fail level than for the distinguished performance scales. Conclusion: These results suggest pass-fail decisions should not be based exclusively on nontechnical skill ratings. We furthermore recommend that regulatory authorities more systematically address interrater reliability in airline instructor training. Airlines as well as training facilities should be encouraged to demonstrate sufficient interrater reliability when using their rating tools.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available