Assessment

Exam Development Summit — Successful Meeting of the Minds

By Emily Yunker, MPA, PMPApril 4, 2019

New EDB members Meredith Quinene, Vivian Moynihan, Marissa Liveris, Jeff Williams, and Peter Fenn learn about the work ahead from Kim Cavanagh (not pictured), joined by PAEA President Jon Bowser. Photo: Emily Yunker

Sunny Phoenix provided an invigorating environment for the fifth annual Exam Development Summit.

In this landmark year, 63 Exam Development Board (EDB) members gathered last month and reviewed a total of 1,472 items for upcoming exams.

The primary purpose of this annual meeting is to conduct the second peer review of exam questions before forms are constructed, reviewed, and published. Each EDB reviews all of that year’s questions, checking for appropriate structure and changes in medical standards in the months since they were written, making edits, and signing off on the final language. This is a major undertaking as each group meticulously reviews between 250 and 400 questions with the guidance of their chairs and in collaboration with editorial consultants.

The work from this meeting will lead to the publication of:

  • Jennifer Forbes, Cody Black, and Anne Schempp (chair of the End of Curriculum EDB) participate in item review. Photo: Emily Yunker
  • Mike Rota, Quincy Jones, and Betsy Ekey of the Internal Medicine and Family Medicine EDB learn about key validation from NCCPA Psychometrician J.B. Weir and Project Manager Evin Robinson. Photo: Emily Yunker
  • Interim CEO Dr. Sara Fletcher giving her keynote address. Photo: Emily Yunker
  • Members of the Emergency Medicine & General Surgery Exam Development Board enjoy lunch in the warm Arizona sun. Photo: Emily Yunker
  • EDS participants compiling kits for service members in partnership with Operation Gratitude. Photo: Emily Yunker
  • The care kits include simple items for service members and hand-written personal notes to thank them and lift their spirits. Photo: Emily Yunker

We also conduct key validation at the Summit facilitated by our psychometric partners. Select volunteers review performance statistics for items administered in the current exams to determine whether they are performing as expected and can be kept, have minor issues and can be tweaked to meet expectations, or have serious interpretation problems and should be deleted.

This is also the primary venue for member development. New members of the EDBs receive their initial training, facilitated by Assessment Editor in Chief Kim Cavanagh. For the training this year, new members had a special visitor – PAEA President Jon Bowser.

As part of our ongoing development efforts for members, keynote speaker Sara Fletcher, PhD, PAEA’s interim CEO, addressed the challenges of aligning assessments to program objectives.

The Summit is also a key opportunity for EDB members — who work together in smaller groups throughout the year — to network as assessment community members. The EDBs represent 53 programs from every region of the country. Members come together not only to “geek out” about assessment, but also to socialize and share what’s going on at their programs.

Each year, we organize a social activity to provide a release for EDB members who work so hard over the four-day meeting. This year, we wanted to do something different by engaging in community service. We partnered with Operation Gratitude, a nonprofit organization that gives back to individuals who serve our country, to compile care kits.

The EDB members, partners, and staff compiled 800 kits. Many found this activity particularly meaningful given the history of the PA profession and its connection to returning service members.

You can learn more about the exam development process, including the additional steps taking place for the End of Curriculum exam over the coming year, on the process page of the assessment website.

We also captured a “day in the life” at the Exam Development Summit in PAEAonline Instagram stories. Check the highlights above the feed to see how our Friday went.

Emily Yunker, MPA, PMP

Emily is director of assessment at PAEA. In this role, she coordinates the development and implementation of the PACKRAT™ and End of Rotation™ exams, working to provide the best possible tools to assess PA student knowledge.