Abstract

Cognitive activity is thought to be a big source of variance in portfolio rating (Van der Shaff et al., 2005). Nevertheless, there has been little research about it and even less about the metacognitive processes involved in the assessment of portfolios. The present research is an exploratory study that attempts to contribute more information about such a topic. It does it by analyzing the metacognitive strategies applied by six novice portfolio raters when applying rubrics to assess teacher performance shown through written portfolio entries. Each rater rated six different portfolio’s entries applying different rubrics of diverse difficulty. Think aloud methodology was used to register the thoughts involved in the 36 rating episodes included in the study. Accuracy of the assessment done by raters is judged by the level of agreement between raters’ judgment and those of the rubric developers. All the think aloud protocols were individually analyzed by two coders and assigned final codes by consensus. Metacognitive capacity is supposed to be highly present in reasoning (Lee, 2004) and facilitate the effective representation of a problem in order to effectively solve it (Gourgey, 1998). It has also been said that metacognition is used more when dealing with more unstructured tasks (Hong, Jonassen y McGree, 2003). The present research tried to explore which metacognitive strategies are applied in the complex task (“problem”) of assessing teacher portfolios, and how the amount and/or type of metacognitive strategies applied by raters relates to the accuracy of judgments involved in the rubric application and the level of clarity (“structure”) of the portfolio entries analyzed. The role of metacognition in lowering assessment bias (Schwarz, et al. 2007) is also explored. A considerable range of metacognitive strategies were found to be used by the raters studied. These were mainly used by the raters to orient, plan and monitor their own thoughts while assessing portfolios. The biggest variety of metacognitive strategies were found to be applied for monitoring cognitive processes and representations, whereas the biggest amount of metacognition had an orientation use. Accurate judgment was found to be more associated to more use of monitoring strategies and less of planning strategies. Less clear portfolio entries required a higher use of metacognition for orientation and less for planning purposes. A specific monitor metacognitive strategy that might lead to lower assessment bias, coded as “consideration of alternative scenarios”, was found to be absent in the rater with the worse performance of the six participant raters.