Total views : 1125

Evaluating the Content Validity of Multistage - Adaptive Tests

Affiliations

  • University of Massachusetts Amherst, United States

Abstract


Validity evidence based on test content is important for educational tests to demonstrate the degree to which they fulfill their purposes. Most content validity studies involve subject matter experts (SMEs) who rate items that comprise a test form. In computerized-adaptive testing, examinees take different sets of items and test "forms" do not exist, which makes it difficult to evaluate the content validity of different tests taken by different examinees. In this study, we evaluated content validity of a multistage-adaptive test (MST) using SMEs' content validity ratings of all items in the MST bank. Analyses of these ratings across the most common "paths" taken by examinees were conducted. The results indicated the content validity ratings across the different tests taken by examinees were roughly equivalent. The method used illustrates how content validity can be evaluated in an MST context.

Keywords

No keywords

Full Text:

 |  (PDF views: 425)

References


  • Adult and Community Learning Services (2005). Massachusetts adult basic education curriculum framework for the English language arts. Malden: MA: Massachusetts Department of Education. Available at http://www.doe.mass.edu/acls/frameworks.
  • American Council on Education. (1995). Guidelines for computerized adaptive test development and use in education. Washington, DC: American Council on Education.
  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
  • Bhola, D.S., Impara, J.C., & Buckendahl, C.W. (2003). Aligning tests with states’ content standards: Methods and issues. Educational Measurement: Issues and Practice, 22(3), 21-29.
  • Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26(2), 44-52.
  • Huff, K. L. & Sireci, S.G. (2005). Validity issues in computer-based testing. Educational Measurement: Issues and Practice, 20(3), 16-25.
  • Kaira, L. T., & Sireci, S. G. (2010). Evaluating content validity in multistage adaptive testing. CLEAR Exam Review, 21(2), 15-23.
  • Luecht, R.M. (2005). Some useful cost-benefit criteria for evaluating computer-based test delivery models and systems. Journal of Applied Testing Technology, 7(2).
  • Luecht, R.M. Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage tests. Applied Measurement in Education, 19(3), 189-202.
  • Luecht, R. M., de Champlain, A., & Nungester, R. J. (1998). Maintaining content validity in computerized adaptive testing. Advances in Health Sciences Education, 3, 29-41.
  • Martone, A., & Sireci, S. G. (2009). Evaluating alignment among curriculum, assessments, and instruction, Review of Educational Research 4, 1332-1361.
  • Messick, S. (1989). Validity. In R. Linn (Ed.), Educational Measurement, (3rd ed., pp. 13-103). Washington, D.C.: American Council on Education.
  • Reese, L.M., Schnipke, D.L., & Luebke, S.W. (1999). Incorporating content constraints into a multi-stage adaptive testlet design (LSAC Computerized Test Report 97-02). Newtown, PA: Law School Admission Council.
  • Sireci, S. G. (1998). The construct of content validity. Social Indicators Research, 45, 83-117.
  • Sireci, S.G., Baldwin, P., Martone, A., Zenisky, A.L., Kaira, L., Lam, W., Shea, C.L., Han, K.T., Deng, N., Delton, J., & Hambleton, R.K. (2008). Massachusetts Adult Proficiency Tests Technical Manual, Version 2. Center for Educational Assessment Research Report No. 677. Amherst, MA: Center for Educational Assessment, University of Massachusetts.
  • Thissen, D. & Mislevy, R.J. (1990). Testing algorithms. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 103-136). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
  • Wainer, H. (1993). Some practical considerations when converting a linearly administered test to an adaptive format. Educational Measurement: Issues and Practice, 12(1), 15-20.
  • Zenisky, A. L., Sireci, S. G., Ryan, A., Baldwin, P., Colvin, K., Li, X., Semerjian, A., & Hambleton, R. K. (2012). The Massachusetts Adult Proficiency Tests technical manual: Version 3. Amherst, MA: Center for Educational Assessment, University of Massachusetts Amherst.

Refbacks

  • There are currently no refbacks.