Demystifying Scale Scores

Co Written by Robert Furter, PhD, MBA, senior director, research and data operations, & Casey Johnson, PhD, senior director of assessment services and psychometrician

 Scale scores can be challenging conceptually for those outside of, or new to, the world of standardized assessment, but it is important to understand the value they bring to making equitable and consistent grading decisions. In this article, we plan to help members by Demystifying Scale Scores, focusing on how scale scores can contribute to informed grading decisions.

In short, scaling is the process of transforming raw test scores (for example, the number of items answered correctly on a test form) to a different metric (for example, the 300-500 scale for End of Rotation™ exams). In doing so, and capitalizing on the underlying process of equating, scale scores are indifferent to the actual test for or set of items a student saw. This is a common practice employed in large-scale, standardized assessment (e.g., AP exams, the SAT or ACT, and GRE), and is employed by PAEA’s End of Rotation™ and End of Curriculum™ exam programs.

When grading students, there is a natural draw to use raw scores. Raw scores are intuitive, direct, and fit nicely into traditional gradebooks. The issue with raw scores is that we cannot disentangle a student’s performance from the difficulty of a test form. Ninety out of 100, or 90 percent, feels like a good grade, just as 50 out of 100, or 50 percent, feels like a bad grade.

However, without knowing the difficulty of the exam form in each case, how certain can we be in interpreting these performances as indicators of knowledge/ability? Or perhaps a less extreme and more common case, let’s assume that a grade of 90 is the dividing line between an A and a B grade – how confident are we that a student with 91 percent deserves an A, and a student with an 89 percent deserves a B, if we are unsure as to which student saw the more difficult test form?

These examples illustrate the ambiguity and potential inequality that can exist when using raw scores. Scale scores, on the other hand, convert each of these raw scores to a common scale that accounts for the difficulty of the exam forms taken. Essentially, scale scores are not dependent on the test form a student took, which allows program faculty to make more direct comparisons between a student’s score and grading criteria or when looking at students’ scores over time.

The benefit of scale scores to make exam results more equitable is tremendous and should not be ignored; however, scale scores in themselves do little to improve the interpretability of scores. At the end of the day, both raw scores and scale scores are simply numbers in need of additional information to make them meaningfully interpretable/usable.

Thanks to the openness and transparency of many PA programs, PAEA has heard from several programs willing to share their current practices for getting from scale scores to grades, a sample of which are published on PAEA’s website. These include norm-referenced interpretations comparing student scores against national reference populations, aligning scaled scores to grade points corresponding to other markers of success in the course/program, and so on.

Regardless of the specific method a program employs in putting exam scores to use (whether those be scale scores or otherwise), the important factors are to be intentional and thoughtful in your practices. Although many of us grew up with 70 percent being a passing grade, and granting the possibility that 70 percent is some sort of universal benchmark, it behooves all assessment developers and users to think critically about how we interpret and use test scores.

In the coming months, PAEA will be publishing “item maps” for each of its EOR exams. The maps will include previously administered EOR items and allow for PA faculty to gain a more concrete understanding of what it takes to achieve varying scale scores, leading to more informed scale score interpretations and uses.

Accompanying the item maps will be extensive documentation on how to use the maps and webinar demo/walkthroughs. Of course, PA faculty are encouraged to reach out to PAEA’s Assessment Services team at exams@PAEAonline.org with questions related to score interpretations or other assessment-related questions. We are here to support you in educating and assessing future generations of PAs!