Abstract

PISA for Schools is an OECD (Organization for Economic Cooperation and Development) program designed to provide results on students' performance in Mathematics, Reading and Science at school level. In order to achieve this purpose, participants are asked to answer a content-based test together with a background questionnaire. Answers are next evaluated by a group of reviewers using a coding guide defined in terms of an ordinal qualitative scale. Although guarantying consistency among reviewers is key, differences may arise on a particular question due to different interpretations of the coding guide for some specific answers. In order to identify the origin of the discrepancies and ensure consistency in the evaluation process of forthcoming editions of the program, this work aims at identifying critical questions that lead to the largest disagreements among reviewers. Ultimately, this critical questions should be examined in detail by performing some qualitative analysis of both the answers provided by the participants and the instructions on the coding guide.