Exam Development — Your Role Along the Way

Exam development is a year-long process that requires everyone’s contributions at various points along the way. One of the most critical contributors to the creation of PAEA exams is you — the subject matter expert (SME). Your clinical knowledge, PA education experience, and skills as an item writer are what qualifies you as an SME. As an SME, there are a variety of times when PAEA relies on your talents. This article will review the steps in the production cycle and the role of our Exam Development Board (EDB) members, and others, along the way.

Item Writing (April – August)

EDB members are asked to write the absolute best items they can from the very beginning of the item writing cycle. The time invested initially has the potential to save time throughout the entire process. This includes following the assigned content and task areas, as well as adhering to the PAEA form and style guidelines and basic item writing principles.

Brief Editorial Review (September)

The editors perform a brief initial review of each new item to check for basic spelling and grammatical errors prior to items being moved to small-group peer review.

Small-Group Peer Review (October)

This step provides for a review of each assigned item individually and as a small group (approximately three members per group). During this stage, the group should agree on suggested edits and/or comments, and the appointed scribe should enter the information into ExamDeveloper. Edits should be made directly in the question as often as possible. ExamDeveloper tracks the changes in the item version, with all versions of the item being preserved for review in the History tab. Comments can be provided to guide the author’s revisions as well.

During this stage, members should be thinking about all aspects of the item — appropriateness of the item, stem content, answer options, and metadata coding. Please see the small-group peer review resources (Peer Review Checklist and Peer Review Comment Guidance document) for more detailed information regarding small-group peer review.

Author Review #1 (November)

Following small-group peer review, each author will receive their items back to review edits and comments from the small-group members. During this process, the authors can accept the edits made by the small group and/or make additional changes based on the comments provided. If an author chooses not to accept the edits or suggested changes, they should be prepared to explain why during large-group peer review.

Editorial Review (January)

The editors perform a detailed review of each item for grammar, spelling, and form and style errors.

Author Review #2 (February 1–15)

After editorial review, the original authors must once again review each item to accept the editorial changes. During this stage, it is important to remember that form and style guidelines must be followed to allow for items within and across the exams to be of one voice. Editorial changes made to follow form and style guidelines should not be rejected unless they make the question incorrect. During this author review stage, the most important focus should be verifying that the edits made do not alter the intention of the item or the medical information.

Item Review Prior to Exam Development Summit (February 18 – March 15)

Prior to the in-person meeting, each EDB member must review each item developed by the group that year and complete the item tracking spreadsheet. This review is designed to make the large-group peer review at the Exam Development Summit (EDS) the most efficient possible. The spreadsheet feedback is used by the EDB chairs and editors to prepare for the in-person meeting. More importantly, the familiarity with each item allows for the most productive in-room discussions and helps to assure all items are appropriately reviewed during the summit.

Exam Development Summit (March 22–25)

The EDS provides a second level of peer review with the full team of SMEs. Active participation in the meeting is expected from all EDB members to allow for the highest quality review, the most thoughtful discussion, and ultimately, the best exams to assess students. While at times intense, the EDS is a wonderful culmination of the work done throughout the year and an impressive display of all members’ knowledge and experience.

Form Review (Summer – dates varies by exam)

The final step before publication is form review, where select SMEs review each item that will be published in that year’s exam(s). The role of the SMEs at this stage is to ensure that item content is accurate and not impacted by any guideline changes, that changes suggested at the EDS were implemented appropriately, and that there is not inappropriate duplication of content within each complete form. Simultaneously, PAEA staff are reviewing each exam form to ensure alignment with form and style guidelines and a unified PAEA voice. This high-level review is an important final sign-off for SMEs, chairs, and staff before the exams are published.

The many steps in the PAEA exam development process are critical to ensure we are providing high-quality, rigorous assessments that speak to the standards we want PA students nationwide to meet. Your role as an EDB member and SME during each stage is important as PAEA works to meet the expectations that you and your peers require. We are so appreciative of the time, knowledge, and commitment that you share with PAEA. The results of your hard work and dedication are evident in the quality of our exams.