Objective multiple-choice exams are commonly used across PA programs to assess students’ knowledge. Thus, test-item writing and exam development are important skills — and often required of PA faculty. One type of test item many particularly struggle with writing is the more challenging, higher-order question, such as the case vignette.
Writing vignettes can be hard, and it certainly takes time, but these types of items result in higher-quality exams that assess students’ capacity for problem-solving and critical thinking, in addition to their medical knowledge. Writing more difficult items does not mean writing questions about obscure topics or adding unnecessary information that tricks examinees; it means writing questions that require students to interpret, analyze, or synthesize information to arrive at an answer. This is what they will do in clinical practice, so it is important to assess this ability in multiple ways, including on objective multiple-choice exams.
Bloom’s Taxonomy is one framework for classification of learning targets. Kim et al. have written a helpful article discussing the use of Bloom’s in the development of multiple-choice items in a pharmacotherapeutics course. PA faculty can use this framework to think about assessing PA students at a variety of levels. The figure below groups some of the domains and provides examples of the types of skills expected at each level, with Remember being the lowest level and Analyze/Evaluate the highest. The example questions below show the difference between questions of a lower order (basic recall) and higher order questions.
Which of the following is the first-line treatment of choice for streptococcal pharyngitis in a patient with no known medication allergies?
- Penicillin VK
- Amoxicillin, clavulanic acid (Augmentin)
- Doxycycline (Doryx)
- Cephalexin (Keflex)
Remember: Recall what the treatment for streptococcal pharyngitis is
This is a one-step thought process in which the student needs only to recall information he or she may have memorized.
An 8-year-old boy presents to the office with acute onset of throat pain for 24 hours. The patient admits to difficulty swallowing, decreased appetite, and a mild nonproductive cough. His mom states he has had a fever at home. Physical examination reveals the nares to be patent without edema. The external auditory canal is patent without erythema; the tympanic membrane is dull with no bulging or erythema. The pharynx is erythematous with tonsillar enlargement and exudates. There is bilateral anterior cervical lymphadenopathy. Lungs are clear to auscultation bilaterally. Which of the following is the most likely diagnosis?
- Bacterial pharyngitis
- Acute rhinosinusitis
- Retropharyngeal abscess
Understand/Apply: Understand the patient scenario to connect to preexisting knowledge — make the diagnosis
This is a two-step thought process in which the student must use information provided to arrive at a diagnosis. This requires connection to previously learned material as the student must interpret the history and physical exam findings provided.
An 8-year-old boy presents to the office with acute onset of throat pain for 24 hours. The patient admits to difficulty swallowing, decreased appetite, and a mild nonproductive cough. His mom states he has had a fever at home. Physical examination reveals the nares to be patent without edema. The external auditory canal is patent without erythema; the tympanic membrane is dull with no bulging or erythema. The pharynx is erythematous with tonsillar enlargement and exudates. There is bilateral anterior cervical lymphadenopathy. Lungs are clear to auscultation bilaterally. Which of the following is the most likely potential complication if this condition goes untreated?
- Splenic rupture
- Periorbital cellulitis
Analyze/Evaluate: Break down information to relate to possible problems/issues; connect ideas in the scenario — most likely complications
This is a multi-step thought process where the student must first arrive at the diagnosis (which is not provided in the vignette) and then determine the most likely complication.
Higher order questions ensure that exams assess more than what students can look up in a book. Writing items with complex scenarios results in the development of exams that better assess clinical reasoning and the ability to apply knowledge.
For more information on item writing, including a step-by-step process, view the PAEA Item Writing module. This includes best practices for the development of vignettes, as well as additional item examples.
Boston University. http://sphweb.bumc.bu.edu/otlt/teachinglibrary/assessment/writingmultiple.pdf
Clay, B. Is this a trick question: a short guide to writing effective test questions. Kansas Curriculum Center. 2001.
Kim, MK, Patel, R, Uchizono, JA, Beck, L. Incorporation of Bloom’s taxonomy into multiple-choice examination questions for a pharmacotherapeutics course. American Journal of Pharmaceutical Education. 2012;76(6) article 14.
National Board of Medical Examiners. Constructing Written Test Questions for the Basic and Clinical Sciences. http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf
Tofade, T, Elsner, J, Haines, S. Best practice strategies for effective use of questions as a teaching tool. American Journal of Pharmaceutical Education. 2013;77(7) article 155.
Vanderbilt University Center for Teaching. https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/
Zimmaro, D. Writing Good Multiple-Choice Exams. Faculty Innovation Center. University of Texas at Austin: 2016.