Advancing Assessment Development: Introducing AI-Supported Item Creation
PAEA is introducing a new approach to support assessment item development, using a proprietary AI system to assist in early-stage item creation.
This work is designed to better leverage the time and expertise of PA educators who contribute to item development, while maintaining the same rigorous standards that define PAEA assessment products.
What’s Changing
PAEA is piloting the use of AI to generate draft items for pretesting on the End of RotationTM and End of CurriculumTM exams.
These items are not scored and are included to collect performance data and evaluate quality, consistent with PAEA’s existing pretest process. Response data from a sample of students allows PAEA to model item performance across ability levels and evaluate how well each question functions.
What’s Not Changing
The core development process remains unchanged.
All items, whether initially drafted by a human or supported by AI, continue to undergo multiple rounds of peer review, psychometric analysis, and validation before becoming scored items.
Supporting Volunteers and Strengthening Item Development
This approach allows item writers to start with draft content rather than a blank page, shifting effort toward reviewing, refining, and evaluating item performance. As Robert Furter, PhD, MBA, PAEA Chief Data Officer and Principal Psychometrician, notes, “This allows us to use our volunteers’ expertise more intentionally – less time starting from scratch, and more time evaluating what makes an item effective.”
This approach supports more efficient use of volunteer expertise and more strategic management of the test bank.
How the Process Works
The AI system is used to draft initial items aligned to PAEA’s specifications and blueprint.
From there, the process remains grounded in expert review. Nationally representative groups of PA educators review, revise, and validate each item. This ensures that items reflect expected levels of student knowledge and align with current educational standards.
Built for Control and Quality
This system is not based on publicly available AI tools. It is closed, proprietary, and developed specifically for PAEA’s assessment program.
All outputs are subject to the same oversight, review, and validation processes already in place.
Looking Ahead
PAEA is currently using this approach for pretest items and will evaluate outcomes over time. If effective, there may be opportunities to expand its use across content areas and stages of the assessment lifecycle.
