Assessment

An Important Change to the PACKRAT and End of Rotation Exams

By Kim Masters, MDJune 1, 2016

Credit: Shutterstock

Credit: Shutterstock

A special task force has been working diligently for the past eight months to ensure PAEA’s exams are DSM 5–compliant.

The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM) was released two years ago in May 2013. Since then, testing organizations have been busy updating their exam items, study guides, and blueprints to be DSM 5–compliant.

The American Board of Psychiatry and Neurology (ABPN) has updated its exams. Accordingly, NCCPA began updating their exams to be DSM 5–compliant at the beginning of 2015. In the instance of PANCE, NCCPA has included the DSM 5 term as the primary term with the DSM IV term following it in parentheses.

ABPN did something similar, providing both the DSM IV and DSM 5 terms for those diagnoses that were changed in name only, expanded into more than one new diagnosis, or subsumed or combined into a new diagnosis. However, they will not be testing on new DSM 5 diagnoses that were not mentioned in the DSM IV. 

After much consideration of these conversion plans, we decided that it was necessary for PA students to know the DSM 5 classifications and criteria, since the PACKRATTM and End of RotationTM exams are specifically meant to test students’ knowledge of the most current information needed to practice medicine.

With the release of version 4 of the End of Rotation exams coming in July, and the subsequent release of version 21 of PACKRAT in November, all of our exams will have only DSM 5–compliant language and diagnostic criteria.

Why Is This Important?

This change is important for a number of reasons:

  • There are several significant differences between the DSM IV and DSM 5. Several diagnoses have been added, removed, or re-classified under different diagnosis categories.
  • It affects every PAEA exam. While it may seem like this shift would only affect the Psychiatry and Behavioral Health End of Rotation exam, every single question in every exam item bank was queried — and updates were made to questions on every exam.
  • The topic lists for the Family Medicine, Emergency Medicine, Women’s Health, Pediatrics, and Psychiatry and Behavior Health End of Rotation exams have been updated and can be found on Endofrotation.org.
  • The topic list and blueprint for the Psychiatry and Behavioral Health End of Rotation exam have undergone significant changes that students and faculty need to be aware of to prepare for taking the exam.
  • It affects students and faculty. Faculty will need to prepare their students for taking exams that are fully DSM 5–compliant, and all study materials for the exams will need to be fully DSM–5 compliant.

The Process

Both PA faculty and students may reasonably ask, “What is so hard about changing terminology to be DSM 5–compliant?”

At first thought, it seems this process would be a simple “find and replace” where the new DSM 5 terms replace the DSM IV terms, but it is actually much more complicated than that. With new diagnostic criteria for many diagnoses and some sweeping changes to diagnosis categories, a systematic review was necessary to ensure the exams are truly DSM 5–compliant. We searched our entire item bank (including all seven End of Rotation banks as well as the PACKRAT item bank) — 833 items were flagged for peer review and 293 items were re-written.

Eight PAs and one MD, the majority of whom are current faculty, were asked in early 2015 to join a DSM 5 Update Taskforce to help make the PACKRAT and End of Rotation test items DSM 5–compliant.

The task force members included:

Carol Galarza, MS PA-C, Wagner College
Natalya Kusheleva, MS, PA-C, Staten Island University Hospital
Jennifer Wegler, MMS, PA-C, University of Southern California
Mark L’Eplattenier, MPAS, PA-C, Hofstra University
Bartley Rust, MPAS, PA-C Rocky Mountain University
Amanda Rahn, MS, PA-C, St. Aloysius Forensic & Mental Health Services
Vanessa Worley, MPAS, PA-C, University of Mount Union
Kimberly Clark, MPAS, PA-C, Marshall B. Ketchum University
Kim Masters, MD, Wake Forest University

These DSM 5 experts undertook a peer review process that spanned eight months, included three stages of peer review, and culminated in a face-to-face meeting of the task force for two days at the PAEA offices in Washington D.C.

Following a systematic search of the PAEA item banks to identify any exam items that may have DSM-relevant information, the peer review process began. In stage one, each content expert was sent about 250 DSM IV questions to review for their relevance and compliance with the DSM 5. Each item included what a student would see on a test:

A Stem (the question), the key (the correct answer), and distractors (the incorrect answers).

For example:

Stem: A 35-year-old married woman presents for the third time in a month to a primary care clinic with vague and persistent pain in her abdomen and chest. She also complains of pain in her hands and legs. She is anxious and worried about these symptoms and has been unable to concentrate on her job as a CPA. She reports that these symptoms have been present for a year, beginning after she was the victim of a sexual assault. The medical work-up and routine laboratory tests have been negative. Her physical exam today, as in the past, is unremarkable except for mild diffuse abdominal pain. The most likely diagnosis for this patient would be:

A. Posttraumatic stress disorder
B. Persistent depressive disorder
C. Illness Anxiety Disorder
D. Somatic Symptom Disorder
E. Factitious Disorder

Key D

Distractors A, B, C, E

In stage one of the review process, the task force was broken into three groups and asked to determine if each question they reviewed was either 1) compatible with the DSM 5; 2) should be re-written to be DSM 5–compliant; or 3) should be discarded because of incompatibility.

In stage two of the review, the individual task force members re-wrote the questions that had been selected for re-write in stage one.

In stage three, the team assembled in person at PAEA’s Washington, D.C. office and for two days reviewed 248 DSM 5 re-written questions. Each question was projected on a screen followed by a discussion of its evolution from the prior DSM IV–compliant item to its current DSM 5 iteration. Included in the discussion were the item’s clarity, relevance, accuracy, current practice, and statistical data.

Moving forward, any re-written questions that appear on future End of Rotation exams will be run through as pre-test (i.e., not scored) items, in order to gather statistics on their performance before they are given as scored, operational items on any exams.

The DSM 5 Taskforce also worked to update all PAEA exam topic lists and blueprints to be DSM 5–compliant. Until the release of the version 4 End of Rotation exams in the first week of July, you will find both DSM IV–compliant topic lists for each of the exams and DSM 5–compliant topic lists on endofrotation.org. This is so students taking version 3 of the exams (the current version) before the first week of July, as well as those taking any of the exams after version 4 is released, can study for the appropriate exams. When version 4 of the End of Rotation exams are released, we will take down the DSM IV–compliant topic lists.

We are committed to making our exam items and materials as up-to-date and relevant as possible to ensure PA educators can confidently deliver our exams to PA students and the next generation of care providers.

PAEA Manager of Assessment and Evaluation Donovan Lessard contributed to this article.

 

 

Kim Masters
Kim Masters, MD

Dr. Masters is a board-certified adult, child, and adolescent psychiatrist with a MD degree from Harvard Medical School. He has been teaching and writing about PA education for over 10 years and is on the volunteer PA faculty of the Medical University of South Carolina and Wake Forest School of Medicine.