Marking criteria calibration involves an iterative process where staff work with samples of student work in order to either create new marking criteria or ensure that draft criteria and standards are appropriate, and are being used appropriately in marking (Prosser, 2014). Essentially, the marking team mark sample assessments, compare marks, discuss discrepancies, and improve the draft criteria and/or move toward consistency in their marking.
A calibration example
McConlogue (2020, p.96) recommends the following process for the development of new marking criteria. This can easily be adapted to incorporate draft criteria as well:
‘To organise calibration sessions, collect exemplars of students’ assignments, particularly around grade boundaries or work that illuminates a particular assessment criterion. Distribute to the teaching team and ask them to read and make a judgement about the assignments and give a rationale for that judgement (without using assessment criteria or rubrics); record these anonymously in a virtual learning environment. Either online or in a face-to-face meeting, discuss
exemplars in groups and through discussion, try to tease out a list of characteristics used by the group to make a judgement about the standard of the exemplar. If agreement is reached, document this, recording the agreed characteristics.
‘If agreement cannot be reached, try to identify the issues that prevented agreement, e.g. did the requirements of the assignment divide opinion? This can happen when students are required to make videos, write blogs and undertake other non-traditional forms of assessment. Assessors may disagree on the relative weighting to assign to the form (e.g. essay, video, web page) and the content of the assignment.
‘Were there disciplinary differences? Did assessors have fundamental, epistemological disagreements? This can happen in inter-disciplinary programmes or in subjects that attract scholars from a variety of disciplines; for example, engineering may have teaching teams drawn from computer sciences, biomedical fields, mathematics and physics…
‘…calibration takes time and needs to be factored into workloads and planned within the programme schedule. There will always be variation in marking standards; there is no one correct grade for complex student work and variation in judgements is to be expected. However, as the studies cited above indicate, calibration has potential to establish more robust academic standards.’
References and resources
Dunn, Lee, et al. (2003) The Student Assessment Handbook: New Directions in Traditional and Online Assessment, Taylor & Francis Group. Chapter 21. https://ebookcentral.proquest.com/lib/durham/detail.action?docID=198508.
McConlogue, T. (2020) Assessment and feedback in higher education: a guide for teachers. London: UCL Press.
Prosser, M. (2014). ‘Perceptions of assessment standards and student learning’, in Advances and Innovations in University Assessment and Feedback: A Festschrift in Honour of Professor Dai Hounsell, edited by Carolin Kreber, et al., Edinburgh University Press.