Four studies, two methods, one accident - An examination of the reliability and validity of Accimap and STAMP for accident analysis

The validity and reliability of human factors and safety science methods are some of the important criteria for judging their appropriateness and utility for accident analysis, however these are rarely assessed. The aim of this study is to take a closer look at the validity and reliability of two systemic accident analysis methods (Accimap and STAMP) by comparing the results of four studies which analysed the same accident (the South Korea Sewol Ferry accident) using two methods. Studies 1 and 2 used Accimap whilst Studies 3 and 4 applied STAMP. The four studies were compared in terms of analysis procedure taken, level of detail, causal factors identified, and the recommendations for improvements suggested by the methods. The results of the causal factor comparison indicate that the reliability (degree of overlap of causal factors identified from the same method, i.e. inter-analyst overlap) of STAMP (65%) is higher than Accimap (38%). The validity (degree of overlap of causal factors identified from two different methods) is as low as 8%. The comparison of recommendations indicates that STAMP-based analyses produce a wider range of recommendations across multiple system levels while Accimap-based analyses tend to focus on whole system-related recommendations. These findings suggest that the use of a more structured method like STAMP can help produce a more reliable accident analysis results.