Abstract

We compared 5 different statistics (i.e., G index, gamma, d=, sensitivity, specificity) used in the socialsciences and medical diagnosis literatures to assess calibration accuracy in order to examine therelationship among them and to explore whether one statistic provided a best fitting general measure ofaccuracy. College undergraduates completed separate 15-item vocabulary, probability, and paper foldingtests by answering a test item and indicating whether or not the item was answered correctly. Wecomputed scores for each of the 5 calibration statistics using the same raw scores for each test andcompared 3 theoretical models, including 1-, 2-, and 3-factor confirmatory factor analysis solutions.Results supported the 3-factor model over the 1-factor and 2-factor models with respect to goodness-of-fit indices and least number of estimated parameters. The 3-factor solution was consistent with thehypothesis that the 5 individual calibration scores are related to 2 different types of 2nd-order processes(i.e., accuracy of judgments about correct and incorrect performance), as measured by sensitivity andspecificity that are subsumed under a general 3rd-order discrimination process as measured by d=.Implications for a theory of calibration accuracy and measurement practice were discussed