Abstract

Validity clearly represents one of the 21st century challenges for the field of assessment in terms of the design and delivery of assessment systems. There is wide consensus in the literature around the importance of validity as a guarantee of quality in any assessment system (Buckingham et al., 1921; Cronbach, 1984; Crooks et al., 1996; Newton, 2012, just to mention a few). However, despite this theoretical recognition, authors like Crooks et al. (1996) have highlighted the way in which procedures related to reliability and generalisation are given more attention in practice than those involved in validity, due to their algorithmic nature, which makes them easier to defend and standardise than the complex and judgement-based approaches required for the construction of a validity argument (Kane, 2008, 2010 and 2011). There is widespread criticism towards the way in which theory around validity is taken into practice by assessment agencies, which is characterised as mainly mechanistic, empiricist, non-holistic and focused on the statistical aspect (Haertel, 1999; Anastasi, 1986; Hubley and Zumbo, 2011; Koch and DeLuca, 2012). Given this long-standing criticism the scarce amount of critical analyses of specific assessment systems and the few attempts to generate new validation procedures seems surprising. The paper contributes to the existing literature by offering a critical analysis of the different dimensions of validity through a concrete case: the System for the Measurement of Quality in Education (SIMCE) in Chile. Given the criticism towards the predominance of empirical and statistical procedures in the practice of assessment validity, a qualitative approach was chosen for the study, where other aspects of validity, relevant to theory but seldom explored in practice, were addressed. On the basis of an initial review of the literature around assessment validity, the following research questions were formulated: Main question: • Is SIMCE a valid assessment process, considering the different dimensions of the concept of validity? Sub-questions: • What are the constructs assessed in SIMCE? Is there a consistent vision about these constructs throughout the system? • What contents does SIMCE assess and how representative is that content of the construct to be assessed? • What do different actors of the process think about the predictive and concurrent dimensions of validity in SIMCE? • Is SIMCE valid for all the purposes and uses currently attributed to it? • Is there a common perspective among actors about the validity of SIMCE and its results? • Is validity maintained throughout SIMCE's processes of production, distribution, correction and use? In order to answer these questions, data were collected considering two sources: 1) 41 documents publicly available in the previous website of SIMCE and the current Agency for Quality website; 2) Semi-structured interviews with 15 key actors of the process related to SIMCE (coordinators, item constructors, item reviewers, among others) and with 20 teachers from a wide variety of profiles. With these data as a basis, the validity of the interpretations that are derived from the results of this assessment process is explored, described, and critically analysed, offering conclusions that are relevant to consider in the development of assessment systems in the future.