The effect of the question mark option in progress testing: a large-scale longitudinal study

Wijk, E.V. van; Donkers, J.; Laat, P.C.J. de; Meiboom, A.A.; Jacobs, B.; Ravesloot, J.H.; ... ; Langers, A.M.J. (2025)

Article / Letter to editor

Introduction: Formula scoring, widely used in medical progress tests (PT), includes a question mark option to discourage guessing, but this feature may disadvantage risk-averse students and bias results due to test-taking strategies. To enhance reliability and more accurately assess ability, Dutch medical schools recently transitioned to a computer adaptive-PT (CA-PT) based on Item Response Theory, which adjusts question difficulty dynamically, excluding the question mark option. This provided a unique opportunity to evaluate the impact of the question mark option in a large cohort. We specifically explored the relationship between question mark use in conventional PT and performance on CA-PT. Methods: Retrospective data from medical students across seven faculties who took both PT formats were analyzed. Z-scores for total score and question mark score (number of unanswered questions) in the conventional PT, and theta score for the CA-PT were assessed. A linear model assessed the effect of the question mark score on theta, corrected for the conventional PT-score. Cluster analysis explored student subgroups per year. Results: Students with similar conventional PT scores who left more questions unanswered on the conventional PT generally performed better on CA-PT. This effect diminished as students advanced through their studies. Cluster analysis revealed a variable effect between different students, most pronounced in year 4, and a reverse effect in year 5. Discussion: Question mark option use significantly impacted student performance on PT, with a remarkable variability among students. This variability suggests that formula scoring captures more than knowledge alone, highlighting the need to align scoring methods with intended assessment goals.

• All authors: Wijk, E.V. van; Donkers, J.; Laat, P.C.J. de; Meiboom, A.A.; Jacobs, B.; Ravesloot, J.H.; Tio, R.A.; Oud, F.M.M.; Kooman, J.P.; Bremers, A.J.A.; Langers, A.M.J.
• Date: 2025-01-01
• Journal: Perspectives on Medical Education
• Volume: 14
• Issue: 1
• Pages: 891 - 904
• DOI: doi:10.5334/pme.1673