Design of peer review

The design of peer review is complex as there are a number of considerations to take into account.  In this website two different components of peer review have been identified – students reviewing the work of peers and receiving reviews from peers. Research is beginning to show that these two components result in different learning benefits. However, in the design of peer review it is usual that the process is reciprocal with students both reviewing and receiving reviews. Hence in the design these two processes need to be considered separately and in combination for best effect.

The following are some principles of good peer review practice constructed by Nicol (2013) and derived from an analysis of the research literature and his own studies of peer review. Please cite this source if you refer to these principles in your publications Nicol, D. (in press) Guiding principles for peer review: Unlocking learners' evaluative skills, In Advances and Innovations in University Assessment and Feedback, eds C. Kreber, C. Anderson, N. Entwistle and J. McArthur, Edinburgh University Press.  The website reference is http://www.reap.ac.uk/PEERToolkit/Design.aspx

Principles of good peer review practice:

  1. Encourage an atmosphere of trust and respect
  2. Use a range of different perspectives for the review tasks
  3. Give practice in identifying quality and in formulating criteria
  4. Require well-reasoned written explanations for feedback responses
  5. Facilitate dialogue around the peer review process
  6. Integrate self-review activities into peer review designs
  7. Encourage critical evaluations of received reviews
  8. Provide inputs that help learners calibrate their judgements

A pre-condition

Recent research shows that many of the learning benefits from producing reviews derive from the fact that students have beforehand written an assignment in the same topic domain as their peers (Nicol, Thomson and Breslin, submitted). Specifically, when the topic domain is the same, evidence shows that students spontaneously use their own work as the initial standard or reference point against which they compare and evaluate the work of peers, even if there are pre-set criteria. This comparative process involves students in reflecting back and thinking about their own work, about its strengths and weaknesses. In other words, it is this comparison that that leads students to rehearse and reconstruct their own understanding of the topic domain. Activating this backward reflection is one of the main purposes for implementing peer review and this would not happen (or would not happen in the same way) by having students merely review a journal paper or another written text. Read more here about this point.. 

A peer review design workshop-toolkit 

A Workshop Toolkit to help academic teams design peer review into their modules and programmes has been developed and piloted.ThisToolkit utilises the principles elaborated above. Specifically, the peer review principles have been turned into conceptual artefacts i.e. cards with a principle and question on one side and examples of implementation on the other side. These artefacts can be used by practitioners to help them think through the design of their peer review implementations. The conceptual artefacts are especially valuable in workshops where the design results from dialogue amongst peers, for example a course team. This toolkit draws on the methodology and the course timeline artefact developed through the Viewpoints project at the University of Ulster. A summary of the evaluation of the Viewpoints project is available here, The peer review artefacts can be found here. 

Short version of these principles

When constructing principles it is important that their meaning is immediately understandable. One way of testing this is to find the shortest expression that still gives the principles meaning as has been done below. This is also useful when formulating tools as in Viewpoints (see above) where the principles become artefacts for course design.  

  1. Encourage an atmosphere of trust
  2. Use a range of perspectives for reviews
  3. Give practice in formulating criteria
  4. Require explanations for feedback responses
  5. Facilitate dialogue around reviews
  6. Integrate self-reviews
  7. Encourage evaluation of received reviews
  8. Help learners calibrate their judgements

 

DESIGN DECISIONS

 

  1. THE GOALS AND FOCUS FOR PEER REVIEW
  2. THE STUDENT ASSIGNMENT  - factual or open-ended (design, essay, computer programme)
  3. UNIT FOR THE PRODUCTION OF THE STUDENT ASSIGNMENT - individual, pairs or group
  4. UNIT FOR THE REVIEW - individual, pairs or group 
  5. MATCHING REVIEWERS - random, by ability, by topic
  6. NUMBER OF REVIEWS - more results in less chance of receipt of poor reviews
  7. PRIVACY - anonymous versus known reviewer and/or author
  8. PEER REVIEW CRITERIA - not given, guidelines given, pre-set teacher-provided 
  9. REVIEW FOCUS - holistic or analytic, the content or the process etc.
  10. USE OF RECEIVED FEEDBACK - drafts, self-review, new task
  11. REQUESTING feedback AND RESPONDING to feedback 
  12. GRADING - no marks, marks for participation, marks for quality of reviews, marks for self-review after peer review

HIGHLIGHTED RESOURCES

 

Main reference paper for principles:

Nicol, D. in press. Guiding principles for peer review: unlocking learners' evaluative skills. In Advances and Innovations in University Assessment and Feedback, ed. C. Kreber, C. Anderson, N. Entwistle and J. McArthur. Edinburgh:Edinburgh University Press. Abstract here

 

Teaching Guide: Using student peer review, Colorado State University link

 

Using peer review to help students improve their writing. Washington University in St.Louis link