“The more we move towards peer anything, the university is going to have to collectively rethink the message that we send to students regarding who holds knowledge, who’s allowed to hold knowledge and who holds the most valuable knowledge and that’s not always the professor.”
–Dr. Catherine Rawn, Senior Instructor in Psychology
Back Story
A few years ago I began experimenting with peer assessments in my introductory PSYC courses—PSYC 101 and 102. These are survey courses including a wide breadth of topic areas within psychology, and they are very large classes (ranging from 250 to 390 students). One of the challenges in these courses is getting any sort of writing in it at all. My other courses include quantitative research methods and statistics for second-year Psychology majors (neither of which uses too much peer assessment), but I have recently pilot-tested a Psychology of Self in Social Media course that involves a lot of peer- and self-assessment. All three of these other classes usually involve around 100 people.
What made you decide to implement peer assessments in your course(s)?
Empirical data suggested that students may improve their own writing skills by reading that of their peers’ and being in that evaluator role. That extra layer of not just writing but having students evaluate appealed to me. There is a richer learning opportunity within the thinking process of evaluation that exposes people to more ideas, to a different lens of viewing written work, which will help them write better. I’d say those are my main motivations. I had a written assignment in those larger intro courses that wasn’t quite working. I was asking students to pick one of three short (600 word) topic papers in which they would take a course concept and apply it anywhere in their lives. Many students would really struggle with writing it, so I tweaked it a few ways. However, it didn’t quite produce the writing that I wanted and it was really tough to get it marked and returned in a timely way with any sort of feedback.
I was first introduced to the idea of peer assessments through Pearson publishing company advertisements. They had partnered with Steve Joordans—a faculty member at University of Toronto who created a platform to facilitate peer assessment in very large classes, called peerScholar. I met Steve at a conference where he was presenting his research on PeerScholar. From then on, peer assessment became something I wanted to experiment with because it seemed like it might help me help students improve their writing.
What were the results?
I kept the basic structure of the assignments (explain a concept and apply it in your life), but made it more frequent and added a layer of peer assessment. Students immediately appreciated reading the examples other people were coming up with. Out of the many students I talked to, although they felt challenged by the more frequent writing assignments, they were able to realize that having exposure to other people’s thinking process and written work was a good thing for their learning. They reported remembering others’ examples, which helped them understand material and recall it as needed.
Whenever I ask students what they thought about peer assessments, the most consistent answer is that it does help them remember the concepts because they’re actively thinking about the multiple ideas they’re exposed to. They are thinking – not just reading over – someone else’s work, but they’re actually thinking about someone else’s work and that can help. Students can say that they understand the concepts better because they are not simply exposed to it, but actually engaging with other people’s work.
It seemed to be working from the learning perspective; the trick was and continues to be the weighting of the peer grade. Students simply don’t trust other students to evaluate their work—Peter Graf and I have survey data from hundreds of students showing this. Based on past research, having 4-5 other students evaluate a paper and then taking the average or median of those marks is actually a pretty good indicator of what that mark should be in most cases. But students don’t trust that. I’ve tinkered with having average peers’ assessments worth more than a few percentage points toward their final grade, and in my experience students have been increasingly dissatisfied as that weight increases. It’s a complexity in using peer review I haven’t quite figured out yet, which leads me to our TLEF project.
Tell us about your TLEF project and what made you decide to do this?
The lack of trust was where Dr. Peter Graf and I converged and how our TLEF project came to life. We trusted that students can evaluate each other’s work but the students didn’t seem to trust each other. We collected some survey data and it confirmed our suspicions – they were essentially saying “Yes, I can do this but my peers can’t”. So whenever we were handing over any portion of their grades to the hands of their peers, they were upset.
We concede that there is variation in student ability, particularly in big introductory psychology classes where we’ve got the fourth year biology majors next to first years who are brand new to campus. Our intent with the project was to create something that would even the initial playing field, create a better sense of a baseline level of competence – which is something we couldn’t guarantee before.
We have developed a multiphase training module in EdX Edge where students are quizzed on the assignment’s rubric and then given sample papers to evaluate. These samples are drawn from past students work (with permission). Students go through a series of phases in the training workshop to practice evaluating based on the rubric criteria. They also receive immediate video-based feedback from their peers and instructor on how well and how accurate they were in using the rubric.
By incorporating this training component, we can confirm that students have at least answered some basic questions about the rubric and used it for a few sample essays. Our hope, although we haven’t analyzed these data yet, is that by recognizing that everybody is going through this training step, students can put more faith in their peers’ review of their work because they know that everyone has gone through this base-level one hour long training. Of course, we’ll also be able to tell whether this workshop actually improves the quality of peer reviews.
What has been the result from implementing this training workshop?
We’re still too early in analyses to be able to reveal results with any certainty—classes just ended, after all. Our research design will enable us to evaluate the effects of having done the training on quality of peer reviews, as well as on perceptions of the peer review process. We will also be able to examine the effect of going through the training at different points of the semester and link that to their quality of writing. Is their writing better after versus before evaluating others’ work? Has their trust in their peers been affected? Those are some of the questions we should be able to answer as we dig in to the data.
What are some of the challenges you’ve faced? Do you have any advice for instructors hoping to implement something similar in their course?
One challenge to anticipate when implementing peer review is resistance from students at the thought of turning their grades over to the hands of their peers. Be very careful and mindful about the proportion of weight that is assigned and ensure you have a clear process for students to be able to dispute the grade that their peers give them if needed. Definitely have multiple peers evaluate a students’ work and take either the average or median of those. It would be fantastic if you have a class size small enough where you can go into every single review to do a check because some students will skip a criterion or not do it at all (that is why in a bigger class it is a little better to take medians rather than averages).
The procedural fairness and the perceived procedural fairness is what makes or breaks this. While students seem to recognize that “I can learn from this”, most of the complaints and worries are about where the grade comes from. Anticipate that fairness will be questioned and challenged and put clear procedures in place where students can dispute their grade. I think it’s fair to be open and ready for that but it’s a challenge at the same time. We believe this training will help students put more faith in the grades they receive from their peers and we think that it will improve the quality of those grades, but that being said, those forefront procedures are essential. Be careful not to weigh it too heavily.
In terms of choosing a peer assessment platform, I recommend speaking with someone who has used it before. Ask to see examples of the user interface, as well as an example output file of the data that the system can export. Be sure it’s in a format you understand and will be able to use to finalize grades.
What are the next steps for your project? What are you planning for year 2?
Complexities aside, I think that there is great merit in peer review of written work. Every time I go back into literature, there seems to be more evidence in the value of this work for learning. It’s something we as professionals are asked to do all the time – evaluate people, evaluate proposals and papers, evaluate award applications – I’m always evaluating someone else’s work and I don’t think that’s something unique to academia. This is a skill that is transferable to many careers so I’m going to keep pursuing a way to build this process into my courses, allowing students to do more writing and reviewing of each other’s work. I want to keep tinkering with it and figure out how to incorporate this in in a way that works, that is fair, and is perceived as fair because I think it’s a really useful skill.
Specifically with respect to this TLEF project, Peter Graf and I are hoping to adapt the training workshop to facilitate peer review of posters, as well as to use as part of TA training for major papers in more advanced courses.
Do you have any final thoughts?
As we end our conversation, I’m thinking about expertise. Peer assessment is one of a whole host of peer-based teaching and learning techniques. The more we move towards “peer-“ anything in teaching, the university is going to have to collectively rethink the message that we send to students regarding who holds knowledge, who’s allowed to hold knowledge and who holds the most valuable knowledge. That’s not always the professor. The assumption that the professor will walk in and be able to know all things and be all things and evaluate everything perfectly is (a) unwarranted, and (b) really challenged whenever we do anything peer-based. To ask students to talk to their neighbor and ask for advice on something generates a response of distrust, because they expect answers from the professor, not their peers. So anything that is peer based pokes at our assumptions about who holds knowledge that is valuable in that classroom. How can we better support students to recognize the value they bring to the task of learning?
