PAEA Assessment Recap at the October 2021 PAEA Education Forum

From October 11-15, we held the PAEA Education Forum. This second virtual Forum was a great success, speakers had plenty to share, and attendees were excited to learn.  

PAEA Assessment hosted a daily booth to address questions from members and held two sessions during the Forum: “PAEA Assessment 411” and “Integrating Cultural Humility and Mindfulness in Test-Item Writing.” 

PAEA Assessment 411  

Assessment 411 provided a number of important updates which are highlighted here. First, we introduced a new member of our team, Robert Furter, PhD. Dr. Furter is the Senior Director of Assessment and a Psychometrician coming to us from the American Board of Pediatrics, and we are thrilled to have him onboard. We also provided a brief update on each of our exam programs.  

PACKRAT® Version 25 has been administered to 219 programs since January 2020. Version 26 will be published November 16, 2021, and you can find the Faculty Guide under Assessment Resources on our webpage. No blueprint or topic list changes were made. All administrations will switch to Version 26 on November 16. Version 25 will remain available as a back-up for retesting purposes, and Version 24 will no longer be available. 

PAEA temporarily waived our proctoring policies for the End of Rotation™ exams effective March 23, 2020, and those policies remain waived today. Based on survey data, we know that about 20% of all programs using the End of Rotation exams are still taking advantage of the waived policy. We have administered End of Rotation exams to 215 programs since January 2020 and anticipate updating policies and publishing new End of Rotation forms in the summer of 2022. We will provide ample advanced notice before any changes are made.  

The PAEA End of Curriculum™ exam has been available since January of 2021. The current performance standards were set in December 2019, prior to exam launch, based on the performance of nearly 800 students at 20 programs who took a pilot version of the exam. In maintaining our commitment to the quality of this new exam, we convened a standard setting panel this August 2021 based on the performance of the over 4,700 students at 130 programs who have taken the exam since launch. We believe that the live administrations more accurately reflect student performance which is why we repeated the panel this year in advance of publishing new forms and standards on January 8, 2022.  

When the new forms are published the score scale of 1200 to 1800 will remain consistent. The score corresponding to Satisfactory Medical Knowledge will also remain the same, at 1400. The score corresponding to Advanced Medical Knowledge was established at 1555, up from 1525 previously (see the sample student performance report below). The scores from previous administrations are comparable to scores moving forward. For example, a score of 1500 on this year’s End of Curriculum exam will be comparable to a score of 1500 next year. This is an important aspect if you are trying to track scores over time. For more information, please review the Faculty Guide under Assessment Resources on our webpage and look for the 2021 Supplement to be published soon.  

Given COVID-19 we have also been extremely vigilant to monitor for any unexplained changes in scores. Recent data looking at overall performance on each of our exams this year compared to pre-COVID administrations, which for End of Rotation and PACKRAT were in 2019 and for End of Curriculum it was in early 2020 (1/6-3/12) is shown below. For End of Rotation exams, there has been a minimal increase between pre-and-post COVID-19 mean student test-taker scores. Both PACKRAT and End of Curriculum scores show no appreciable change. When comparing past versions within the same time frames, there were also similar, minimal increases in student test-taker means year over year. While we cannot be certain the reason for the increases in exam scores, analysis indicates negligible impact and valid exam results.  

Update on the PAEA End of Rotation “General Surgery” Blueprint 

The ARC-PA Accreditation Standards for Physician Assistant Education, 5th edition modified the requirement that “supervised clinical practice experiences should occur with preceptors practicing in the general surgery discipline (4th edition B3.07)” to “supervised clinical practice experiences must occur with preceptors who enable students to meet program-defined learning outcomes for surgery (5th edition B3.07.)” 

This change affects one of the core rotations for which PAEA provides an End of Rotation exam. The PAEA End of Rotation General Surgery Blueprint Workgroup meet his past year to determine how best to access the surgical knowledge of our students. The workgroup did a literature review and conducted a qualitative program survey. There are still several milestones pending before changes to the current General Surgery Blueprint are announced. Some of the expected changes include: 

• A professional practice task area will replace health maintenance. 
• There are three new proposed content areas: breast surgery, pain medicine/anesthesia, and trauma/acute care. 
• There are new perioperative setting targets. 
• Adjustments to content distribution by content area, notably a reduction in the overall number of gastrointestinal/nutritional questions on the exam. 

EthicsPoint Release  

PAEA has launched a new tool that will allow secure, confidential reporting of suspected inappropriate exam behavior on PAEA Assessment exams to help support the integrity of the exams for all stakeholders. This tool, PAEA EthicsPoint, includes an online form and a toll-free hotline using a third-party service that will securely collect critical information so that PAEA can conduct a thorough evaluation of each report. PAEA EthicsPoint is available to students, faculty, and staff who may have information about potentially inappropriate collection, dissemination, or use of exam information or inappropriate behavior related to PAEA Assessment exams. 

PAEA Assessment Policies have been updated to reflect the new procedures. What constitutes a violation and the responsibility to report to PAEA have not changed. 

The Next Generation Platform 

Thanks to member feedback, we continue to make improvements to ExamDriver. Recently, this includes improvements to server speed and stability as well as testing new accessibility features. As a reminder, Microsoft is also ceasing Internet Explorer operation and focusing on Microsoft Edge as its sole web browser. PAEA is ending support for Internet Explorer on ExamDriver on November 9, 2021, so we can focus support on compatible servers.  

In addition, we officially announced our goal to modernize the PAEA Assessment platform. This past winter we surveyed members for an understanding of their needs and to provide data to assist us as we work to design and implement new features that our members prioritize. Our plan is to begin the launch of the new platform in the summer of 2022. You can expect frequent communications from our team well in advance and leading up launch.  

 Integrating Cultural Humility and Mindfulness in Test-Item Writing 

One of the most important topics during the 2021 Forum was cultural diversity and inclusion. To complement this, PAEA Assessment provided a session on how bias is manifested in test item writing. If you missed this session or want to take a deeper dive into the topic register for our three-hour workshop in January 2022.  

Please contact exams@PAEAonline.org with any questions.