Embracing Our Visual Inspection and Analysis Tradition: Graphing Interobserver Agreement Data

Most investigators using single-case experimental designs use interobserver agreement (IOA) checks to enhance the credibility of the collected data, and they report the results of those assessments using percentage of agreement estimates. An alternative is to graph both observers' records of the measured behavior on the primary study graphs. Such graphing leads to greater transparency and is advocated for five reasons: (a) to make explicit how IOA assessments were distributed across the study, (b) to ensure agreement estimates are reported at the level of the measured behavior of interest rather than a broader observational code, (c) to detect observer drift, (d) to detect the effect of observer expectations, and (e) to put the IOA data in a more suitable context for assessing the internal validity of the study by eliminating the need for an arbitrary agreement criterion.