Effective Small Group Peer Review
One of the many strengths of the PAEA Assessment products for programs is the multi-stage peer review process. Peer review provides content validity – assurance of the appropriateness and accuracy of the exam content as decided by subject matter experts in PA education. An item writer authors items through one lens, and peer review adds multiple, objective points of view to improve the quality of the item in collaboration with the item writer. Even more, peer review results in an exam that has seemingly been written in one voice, not 12 to 14 individual ones.
The PAEA peer review process begins with small group peer review. During this stage, groups of three or four Exam Development Board members review a subset of items written by their colleagues. The goal of this review is to provide a first critical eye on the content of the items. The success of this process is dependent upon the detail of the review, as well as the specificity of the feedback, so that the author can respond accordingly and make necessary changes.
During the small group peer review, each member of the peer review team should review each question assigned to the group in ExamDeveloper. The group should then schedule a call to discuss each of the items. One member of the peer review team should serve as the scribe. The scribe will make suggested edits to the items (which can be done directly in the item in ExamDeveloper) and/or record specific comments (feedback, questions, or options for revision) in ExamDeveloper. After small group peer review, the original author will have the opportunity to review the edits and comments made by the review team.
The more specific the feedback, the easier it will be for the original author. Remember: the author thought the item was truly good when he/she first submitted it, so saying “the stem needs to be reworked” or “the stem needs more detail” does not help. The group should think about ways to rework the stem and make those edits, or lay out a direction for the author. Reviewers should give suggestions of details to add, and/or offer alternative medication or test options. It is the responsibility of the small group to communicate their concerns if the group feels something provided in the original item is inappropriate.
Another benefit to providing a greater level of detail at this stage is that it will make the large group review at the Exam Development Summit more efficient. The large group review should be the final review and approval of items – a sign-off process. The large group is not the best forum for a finely detailed editing of items, or fixing content or coding. Ideally, that should all be taken care of during the small group review and the subsequent medical editing process, as well as these stages’ respective author reviews.
A checklist and comment guide has been developed to assist the small group peer reviewers. These resources provide consistency to the process and allow quick reference during the review. Group members should have these resources available during both individual review and during the group call. Below is an example item, followed by suggested feedback that could be provided by small group peer review.
EXAMPLE:
Original item submission:
Possible edits/comments during small group peer review:
The final question would appear as:
