Testing, Testing, 1, 2, Version 3

Beginning Wednesday, July 8, all students taking an End of Rotation exam will automatically see the new enhanced version. It’s taken a lot of blood, sweat, and tears from our amazing PAEA volunteer content experts and PAEA staff alike to get to this point (okay, maybe not blood).

These annually updated exams, written and peer-reviewed by your fellow educators in the PA community, are edited and statistically validated by national test editors and psychometricians. And the feedback that you — our members — have provided has been invaluable in helping us make enhancements to our system to offer the best exam-taking experience possible.

Here are 5 things you should know about the new release and what it means for you and your students:

1. Updated national data — and more of it

Like last year, national comparative data will be available at the time of Version 3’s release. This data continues to be based on a representative sample of 750–1,000 PA students who have taken these exams previously. In past years, the national data we made available consisted of the national average and standard deviation for each form of each exam.

This year, we’re taking it a step further to help you and your fellow faculty feel confident in the quality, consistency, and difficulty of our End of Rotation exams. In the coming weeks, we will be releasing more detailed technical information about the new data. So stay tuned and visit endofrotation.org after the release for more information.

2. The one-hour block is gone!

We heard your frustration about the system’s forced one-hour block between scheduling an exam and being able to administer it. Working with our partners at NCCPA, we made the necessary changes, so that now you can schedule an exam for a student to start almost immediately.

The previous inability to edit an exam event less than one hour before that exam’s start time will also no longer be an issue. From now on, you will be able to add or edit exam events up to — and even after — an exam has started.

Up until recently, when scheduling students to take multiple End of Rotation exams on one day, you couldn’t let students start an exam earlier than the scheduled start time, so if a student finished an exam early and wanted to get started on their second exam, they had to wait. But now proctors are able to immediately start a student’s second exam after they finish the first, even if it’s a few hours earlier than the originally scheduled exam start time. Not only does this save time for you and your students, but it also allows you to better meet individual test-taker’s needs, so that those who are in the “zone” can continue right on to their next exam.

3. Improved non-proctored exams

We realize that only PACKRAT can be given as a non-proctored exam and that the new version of PACKRAT doesn’t come out until November, but we wanted to make some enhancements now to address faculty feedback about the non-proctored experience.

We all know that computers don’t always cooperate, and that can be problematic when students are taking an exam from home. In the past, if they were accidentally kicked out of their exam, it was difficult to find a faculty member who could reauthorize them to get back in.

We now have changed our system to allow students taking a non-proctored PACKRAT exam to reauthorize themselves, if needed. They are able to do this up to three times and only if it has been less than 10 minutes since they were last in the exam.

We also now require faculty to provide primary and secondary points of contact for students when scheduling the PACKRAT exam. So if students have problems or questions during the exam, they know whom to contact for assistance.

4. Information to help with accountability

We’ve all heard of Murphy’s Law and during an exam administration all sorts of things can happen. So wouldn’t it be helpful to have a report that verified for you what was going on for each of your students during an exam? Based on your feedback, we developed the Exam Activity Report, now available to users within our exam system.

For each exam a student takes, proctored or non-proctored, an activity report is generated and available both during and after the exam. The report will show:

• Scheduled start time
• Actual start and completion times
• Number of test items seen and answered
• Total time given, total time used, and total time remaining
• Events (along with the time of the event) captured during the exam to include: disconnection, reauthorization, lost focus

If a student is concerned that they weren’t given the correct amount of time for an exam, you can refer to this report and say with confidence whether or not the system gave them the full time. Or, if you suspect a student of cheating during an exam, you can use the exam event data on the activity report to help back you up. And if you want to check in and see how your students’ non-proctored PACKRAT went, you can look at their individual activity reports to see what occurred.

5. Student score reports

Finally, after discussions with our partners and psychometricians at NCCPA, we decided to make a few minor changes to the student score reports. In the past, students have been given a report with a breakdown by content and task areas, percentage correct in those areas, and their national percentile rank. Because this percentile data can be misleading to students, we have changed how we report a national comparison for students to better align with best practices in the national testing industry.

Different content areas make up varying proportions of our exams, so sub-scores reported out for specific content categories and task areas are subject to misinterpretation. This can lead to students placing undue weight on their percentile ranking in a content area comprising just 3% of the exam versus a content area that comprises 50% of the exam. And even if a student does accurately interpret their percentile rankings, it still is not the optimal way for them to see how they performed nationally relative to their peers. That is why, starting with End of Rotation Version 3, the student score reports will contain the national average performance overall and for each content category and task area.

At PAEA, we constantly strive to ensure both our exams and our exam system are fulfilling your needs. These enhancements are the result of the candid feedback we have received from our loyal exam users. That partnership, along with our invaluable volunteer content experts, will continue to allow us to offer you the best possible assessment tools we can.

If you have any questions or comments on PAEA’s End of Rotation or PACKRAT exams, please contact our exam support team at exams@paeaonline.org.