Total views : 1007

Computer Adaptive Testing for Small Scale Programs and Instructional Systems

Affiliations

  • Graduate Management Admission Council

Abstract


This study investigates measurement decision theory (MDT) as an underlying model for computer adaptive testing when the goal is to classify examinees into one of a finite number of groups. The first analysis compares MDT with a popular item response theory model and finds little difference in terms of the percentage of correct classifications. The second analysis examines the number of examinees needed to calibrate MDT item parameters and finds accurate classifications even with calibration sample sizes as small as 100 examinees.

Full Text:

 |  (PDF views: 418)

References


  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F.M. Lord & M.R. Novick (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
  • Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
  • Cover, T.M., & Thomas, J.A. (1991). Elements of information theory. New York, NY: Wiley.
  • Cronbach, L.J., & Gleser, G.C. (1957).Psychological tests and personnel decisions. Urbana, IL: University of Illinois Press.
  • Domingos P., & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29, 103-130. Retrieved from http://citeseer.nj.nec.com/48.html.
  • Eggen, T. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23(3), 249-61.
  • Hambleton, R., & Novick, M. (1973). Toward an integration of theory and method for criterion-referenced tests. Journal of Educational Measurement, 10, 159-170.
  • Huyhn, H. (1976). Statistical considerations for mastery scores. Psychometrika, 41, 65-79.
  • Kingsbury, G. G., & Weiss, D. J. (1981). A validity comparison of adaptive and conventional strategies for mastery testing. (Research Report 81-3). Minneapolis, MN: University of Minnesota, Department of Psychology, Psychometric Methods Program.
  • Kingsbury, G.G., & Weiss, D.J. (1979). An adaptive testing strategy for mastery decisions. (Research report 79-05). Minneapolis: University of Minnesota, Psychometric Methods Laboratory.
  • Kingsbury, G. G., & Weiss, D. J. (1983). A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 257-283). New York,NY: Academic Press.
  • Kullback, S., & Leibler, R.A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22, 79-86.
  • Lewis, C., & Sheehan, K. (1990). Using Bayesian decision theory to design a computerized mastery test. Applied Psychological Measurement, 14(2), 367-86.
  • Lin, C.-J., & Spray, J. (2000). Effects of item-selection criteria on classification testing with the sequential probability ratio test. (Research Report 2000-8).Iowa City, IA: ACT, Inc.
  • Macready, G., & Dayton C. M. (1992). The application of latent class models in adaptive testing. Psychometrika, 57(1), 71-88.
  • Mislevy, R. J., & Gitomer, D. H. (1996). The role of probability-based inference in an intelligent tutoring system. User-Mediated and User-Adapted Interaction, 5, 253-282.
  • Reckase, M. D. (1983). A procedure for decision making using tailored testing. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 237-255). New York, NY: Academic Press.
  • Rudner, L.M. (2002, April). An examination of decision-theory adaptive testing procedures. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
  • Rudner, L.M. (2009). Scoring and classifying examinees using measurement decision theory. Practical Assessment, Research & Evaluation, 14(8). Retrieved from http://pareonline.net/getvn.asp?v=14&n=8 .
  • Shannon, C.E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379-423, 623-656. Retrieved from http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html
  • Sheehan, K., & Lewis, C. (1992). Computerized mastery testing with nonequivalent testlets. Applied Psychological Measurement, 16, 65-76.
  • Spray, J. A., & Reckase, M.D. (1996). Comparison of SPRT and sequential Bayes procedures for classifying examinees into two categories using a computerized test. Journal of Educational and Behavioral Statistics, 21(4), 405-14.
  • Thompson, N. A. (2007). A Practitioner’s Guide for Variable-length Computerized Classification Testing. Practical Assessment Research & Evaluation, 12(1). Retrieved from http://pareonline.net/getvn.asp?v=12&n=1 van der
  • Linden, W. J., & Mellenbergh, G.J. (1978). Coefficients for tests from a decision-theoretic point of view. Applied Psychological Measurement, 2, 119-134.
  • Vos, H. J. (1999). Applications of Bayesian decision theory to sequential mastery testing. Journal of Educational and Behavioral Statistics, 24(3), 271-92.
  • Wald, A. (1947). Sequential analysis. New York: Wiley. Welch, R.E., & Frick, T. (1993). Computerized adaptive testing in instructional settings. Educational Technology Research & Development, 41(3), 47-62.

Refbacks

  • There are currently no refbacks.