4.5 Article

Pass/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes

Related references

Note: Only part of the references are listed.
Article Education & Educational Research

Re-conceptualising and accounting for examiner (cut-score) stringency in a 'high frequency, small cohort' performance test

Matt Homer

Summary: The study found that examiners are the main source of variation in cut-scores in OSCE examinations, accounting for 56% of the variance, compared to the smaller effects of stations and exams. Aggregating data to the exam level tends to ameliorate the impact of examiner stringency on cut-scores.

ADVANCES IN HEALTH SCIENCES EDUCATION (2021)

Article Education, Scientific Disciplines

Examiners' decision-making processes in observation-based clinical examinations

Bunmi S. Malau-Aduli et al.

Summary: This study explored examiner decision making during exit-level OSCEs and found that examiners experience higher cognitive workload when assessing borderline candidates. Five major themes were identified in examiner decision making. Examiner training and calibration are crucial for improving accuracy and consistency in judgements.

MEDICAL EDUCATION (2021)

Article Surgery

Implementation of the ACS/APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory

Kristen M. Jogerst et al.

Summary: Rater reliability for assessing hand-sewn bowel and vascular anastomoses using OSATS GRS was found to be inconsistent, indicating the need for rater training to improve scoring agreement and reduce variability in technical skill assessments.

AMERICAN JOURNAL OF SURGERY (2021)

Article Education, Scientific Disciplines

Conjunctive standards in OSCEs: The why and the how of number of stations passed criteria

Matt Homer et al.

Summary: This paper explores the motivations behind the adoption of the MNSP in OSCEs and introduces potential methods for setting an examinee-centred standard. It discusses the psychometric arguments for and against limiting compensation in OSCEs, emphasizing the value many stakeholders place on an MNSP standard. The paper also highlights the need for more empirical work to understand the impact of different standard setting methods on pass/fail decision-making in high stakes settings.

MEDICAL TEACHER (2021)

Article Education, Scientific Disciplines

Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE)

Peter Yeates et al.

Summary: It is important to ensure that examiners in different circuits judge OSCEs to the same standard, as recent research shows that examiner-cohorts can significantly impact candidates' outcomes. A method called VESCA has been developed to address examiner-cohort effects and enhance the quality assurance of distributed or national OSCEs. This study found that video-based scoring comparison and adjustment can improve the fairness and validity of exams.

ACADEMIC MEDICINE (2021)

Article Education, Scientific Disciplines

Building reliable and generalizable clerkship competency assessments: Impact of 'hawk-dove' correction

Sally A. Santen et al.

Summary: This study examined the impact of raters' differences in medical student workplace-based competency assessments, finding that most variance in assessment scores was attributed to raters rather than students. Making adjustments for rater stringency-leniency improved the psychometric characteristics of assessment scores.

MEDICAL TEACHER (2021)

Article Education, Scientific Disciplines

Are rating scales really better than checklists for measuring increasing levels of expertise?

Timothy J. Wood et al.

MEDICAL TEACHER (2020)

Article Education, Scientific Disciplines

Shining a spotlight on scoring in the OSCE: Checklists and item weighting

Matt Homer et al.

MEDICAL TEACHER (2020)

Article Education, Scientific Disciplines

Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs

Peter Yeates et al.

MEDICAL EDUCATION (2019)

Article Mathematical & Computational Biology

Using simulation studies to evaluate statistical methods

Tim P. Morris et al.

STATISTICS IN MEDICINE (2019)

Article Social Sciences, Mathematical Methods

Using the Linear Mixed-Effect Model Framework to Estimate Generalizability Variance Components in R

Zhehan Jiang

METHODOLOGY-EUROPEAN JOURNAL OF RESEARCH METHODS FOR THE BEHAVIORAL AND SOCIAL SCIENCES (2018)

Article Education & Educational Research

The attenuation of correlation coefficients: a statistical literacy issue

David Trafimow

TEACHING STATISTICS (2016)

Article Computer Science, Interdisciplinary Applications

Fitting Linear Mixed-Effects Models Using lme4

Douglas Bates et al.

JOURNAL OF STATISTICAL SOFTWARE (2015)

Review Education, Scientific Disciplines

A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment

Jonathan S. Ilgen et al.

MEDICAL EDUCATION (2015)

Article Education, Scientific Disciplines

How to set standards on performance-based examinations: AMEE Guide No. 85

Danette W. McKinley et al.

MEDICAL TEACHER (2014)

Review Education & Educational Research

Exploring the impact of mental workload on rater-based assessments

Walter Tavares et al.

ADVANCES IN HEALTH SCIENCES EDUCATION (2013)

Article Education, Scientific Disciplines

The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: Organisation & Administration

Kamran Z. Khan et al.

MEDICAL TEACHER (2013)

Article Medicine, Research & Experimental

A method for identifying extreme OSCE examiners

Ilona Bartman et al.

CLINICAL TEACHER (2013)

Article Education, Scientific Disciplines

Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68

Ralph Bloch et al.

MEDICAL TEACHER (2012)

Article Education, Scientific Disciplines

How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49

Godfrey Pell et al.

MEDICAL TEACHER (2010)

Article Education, Scientific Disciplines

The practical value of the standard error of measurement in borderline pass/fail decisions

Richard Hays et al.

MEDICAL EDUCATION (2008)

Article Education & Educational Research

Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEs

Peter H. Harasym et al.

ADVANCES IN HEALTH SCIENCES EDUCATION (2008)

Article Education, Scientific Disciplines

How specific is case specificity?

G Norman et al.

MEDICAL EDUCATION (2006)

Review Psychology, Applied

The sources of four commonly reported cutoff criteria - What did they really say?

CE Lance et al.

ORGANIZATIONAL RESEARCH METHODS (2006)

Article Education, Scientific Disciplines

Comparison of a rational and an empirical standard setting procedure for an OSCE

A Kramer et al.

MEDICAL EDUCATION (2003)

Article Education, Scientific Disciplines

AMEE Guide No. 18: Standard setting in student assessment

MF Ben-David

MEDICAL TEACHER (2000)