Measuring the fit between human judgments and automated alerting algorithms: A study of collision detection

Methodologies for assessing human judgment in complex domains are important for the design of both displays that inform judgment and automated systems that suggest judgments. This paper uses the n-system lens model to evaluate the impact of displays on human judgment and to explicitly assess the similarity between human judgments and a set of potential judgment algorithms for use in automated systems. First, the need for and concepts underlying judgment analysis are outlined. Then the n-system lens model and its parameters are formally described. This model is then used to examine a previously conducted study of aircraft collision detection that had been analyzed using standard analysis of variance methods. Our analysis found the same main effects as did the earlier analysis. However, n-system lens model analysis was able to provide greater insight into the information relied upon for judgments and the impact of displays on judgment. Additionally, the analysis was able to identify attributes of human judgments that were - and were not - similar to judgments produced by automated systems. Potential applications of this research include automated aid design and operator training.