Assessment

End of Rotation Exam Scale Score FAQs

June 13, 2018

Starting with Version 6, End of Rotation™ exam scores will be reported on a scale from 300 to 500. Using this standard metric will allow for easy comparisons between versions of the exam over time and forms of the exam delivered to a cohort. In the course of webinars about scale scores delivered to faculty over the last month, a number of questions have been asked that deserve some detailed, written explanation. We will be answering those questions over the next two months in this FAQ series.

Here is your introduction to scale scores and what they mean for the End of Rotation exams.

What will the score reports look like?

The score reports will look very similar to the reports members currently access on ExamDriver. There will be an individual student performance report, a program-level performance report, and a program-level composite Excel spreadsheet.

The cohort composite report. The composite report will look almost identical to the current one. The main difference is that you will now see scale scores instead of raw scores for Version 6 and later. Version 6 will appear on a separate Excel worksheet from earlier exam versions in order to separate exams that are given a raw score from exams that are given a scale score. These are denoted by exam administration date (arrow A in the image below). We will still display the form numbers of each exam so that you can be sure students are seeing a different exam if they are retested (arrow B in the image below).

Here is an example of the cohort composite report:

For most of the exams, the key is the same. Things get a little more complicated for the Psychiatry and Behavioral Health exams. Since we changed the blueprint with Version 4 to be compliant with DSM-5, we had to add an additional set of tabs. So now, there will be third Class Scores tab (arrow A in the image below). The second key applies to both later tabs.

The program performance report. Moving to a scale score means the forms on each specialty exam are equated, so we will no longer need to provide program-level performance reports for each individual form — which is an exciting enhancement. From now on, you will only see one performance report for each exam version. In other words, where you saw FMEOR v5.1 and v5.2 performance reports before, you will only see FMEOR v6 moving forward. The main difference you will notice, other than the scores,  is in the title of the columns (arrow A in the image below) moving from % Correct to Average Score and National Average. Also, the Number of Exams ticker (arrow B in the image below) will move up a little faster because it includes both forms.

Here is an example of the new program performance report:

The report shows the average score for all students in the class, as well as the national statistics for students from all programs who have answered these questions in previous versions.

There is a new disclaimer at the bottom of the report that clarifies the parameters of the scale and the national reference population used to create the national average. It states:

Comparative data contained in both the class average table at the top of the report and the “Content Area Feedback” sections are based on a reference population of test takers. The value listed in the “Class of <class name from ExamDriver> Average Score” column reflects your cohort’s performance on the test questions in each content area. Like the total score, your program’s performance ranges from 300 to 500.

The “All PROGRAMS National Average” column is comparative data. The score in this column is the average score of a reference population of test takers from all PA education programs on items in each content area on the <specialty of the exam> End of Rotation examination. The information provided in the last column is provided as a point of reference so you can compare your cohort’s performance to national performance.

For guidance on how to interpret the results, please visit EndofRotation.org.

The student performance report. The student performance report will look similar to the previous reports, but with a new way to display overall score and different subheadings. It will give the student’s total scale score at the top on a visual performance bar (arrow A in the image below). Next, content area and task area feedback will be reported with the student’s scale score and the national average score (arrow B in the image below). As with previous student reports, keyword feedback will be at the bottom of the page (not shown — it didn’t change).

Here is an example of the new student report:Family medicine student report
There is a new disclaimer at the bottom of the report that clarifies the parameters of the scale and the national reference population used to create the national average in the content area section. It states:

 

The comparative data in the “Feedback by Content Area” section are based on a reference population of test takers. The listed percentages in the “Content Blueprint” column reflect the percentage of questions targeted for inclusion on each exam form; however, the actual content percentage on your test form may vary slightly from this target.

The number listed in the “Your Performance” column reflects your performance on the test questions in each content area. Like the total score, your performance in each content area ranges from 300 to 500.

The “National Performance Average” column is comparative data. The score in this column is the average score of a reference population of test takers from all PA education programs on items in each content area on the <specialty of the exam> End of Rotation examination. The information provided in the last column is provided as a point of reference so you can compare your performance to your peers.

What will the conversion tool look like?

To make the enhancement to scale scores as seamless as possible, an online conversion application will be available in ExamDriver on June 27. The conversion tool will be similar to conversion calculators available on other apps, such as those that are commonly used to calculate body mass index and medication dosages. The scale score from the student report or composite report can be entered into a column and the raw score will be computed, and vice versa.

Here is what the conversion tool will look like:

 

As you can see from the above screen shot, you will be able to pick the specialty exam (arrow A), version of the exam (arrow B), and exam form (arrow C), and covert the raw number to a scale score by entering in the data as they appear on the score report starting with the total correct (arrow D). It also works in reverse: programs that are deep into their clinical year and need to continue providing raw scores will be able to look those scores up as well. Based on your feedback, we are currently investigating mass conversion options for these programs as well. We intend to have this available in the next month.

The conversion tool will be available to faculty only in ExamDriver, using a link in the upper right corner of the Reports tab.

Does the enhancement change what the Reports tab will look like?

The reports tab will look slightly different moving forward. When you select an End of Rotation exam from the Filter to Exam dropdown, a little link will appear on the right side called Historical Score Lookup. This link will take you to the score conversion tool. Also, you’ll only see one performance report for all Version 6 exams rather than two.