Total views : 1965

Combining the best of Two Standard Setting Methods: the Ordered Item Booklet Angoff


  • Alpine Testing Solutions , 51 West Center Street, #514, Orem, UT 84057, United States


This article describes a hybrid standard setting method that combines characteristics of the Angoff (1971) and Bookmark (Mitzel, Lewis, Patz&Green, 2001) methods. The proposed approach utilizes strengths of each method while addressing weaknesses. An ordered item booklet, with items sorted based on item difficulty, is used in combination with individual item ratings rather than a bookmark placement. The ordered item booklet attempts to make individual item ratings more palatable by providing some amount of information about item difficulty to the Subject Matter Experts (SMEs) participating in the standard setting. Allowing for individual item ratings attempts to addresses the issue of SMEs disagreeing with the order of items in the booklet in a Bookmark standard setting. This article illustrates the method as it was applied to a healthcare examination used for state licensure, comparing it with a more traditional Angoff approach. Results suggest that the proposed method has promise for future use pending further exploration.


Angoff Method, Bookmark Method, Standard Setting

Full Text:

 |  (PDF views: 1075)


  • Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). Washington, DC: American Council on Education.
  • Bishop, N. S., Nhouyvanisvong, A., Zhu, X, & Liassou, D.(2011). Establishing Cut Scores on a Modified Assessment for NCLB Using an Ordered-Item Angoff Standard Setting Methodology. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
  • Buckendahl, C. W., Smith, R. W., Impara, J. C., & Plake, B. S. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39, 253–264.
  • Cizek, G., & Bunch, M. (2007). A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage Publications.
  • Davis-Becker, S. L., Buckendahl, C. W., & Gerrow, J. (2011). Evaluating the Bookmark standard setting method: The impact of random item ordering. International Journal of Testing, 11(1), 24–37.
  • Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. Brennan (Ed.),Educational measurement (4th ed., pp. 433–470). Westport, CT: Praeger Publishers.
  • Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 355–368.
  • Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 485–514). New York, NY: American Council on Education and Macmillan.
  • Karantonis, A., & Sireci, S. (2006). The bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4–12.
  • Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The Bookmark procedure: Psychological perspectives. In G. Cizek (Ed.), Settingperformance standards: Concepts, methods, and perspectives (pp. 249–282). Mahwah, NJ:Lawrence Erlbaum.
  • Pellegrino, J. W., Jones, L. R., & Mitchell, K. J. (1999). Grading the nation’s report card. Washington, DC: National Academy Press.
  • Peterson, C. H., Schulz, E. M., & Engelhard Jr., G. (2011). Reliability and validity of bookmark-based methods for standard setting: Comparisons to Angoff-based methods in the National Assessment of Educational Progress.
  • Educational Measurement: Issues and Practice, 30(2), 3–14. Schulz, E. M., & Mitzel, H. (2005, April). The Mapmark standard setting method. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, QB.
  • Shepard, L., Glaser, R., Linn, R., & Bohrnstedt, G. (1993). Setting performance standards for student achievement. Stanford, CA: National Academy of Education.
  • Skaggs, G., & Tessema, A. (2001, April). Item disordinality with the Bookmark standard setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.


  • There are currently no refbacks.