Evaluating Item Selection Methods for Adaptive Tests with Complex Content Constraints

Abstract

Adaptive testing designs have become go-to methods for large-scale test administration due to their ability to provide more accurate scores with fewer items. In recent years, new designs have been introduced, such as on-the-fly multistage testing (OMST), that combine the advantages of the well-established computerized adaptive testing (CAT) and multistage testing (MST) designs. While adaptive testing has attracted a tremendous amount of research, most studies have used only one set of test specifications to constrain the content of the test. Through Monte Carlo simulation, this study evaluated the effectiveness of CAT, MST, and OMST under varying levels of test specification complexity. Specifically, the constrained item selection methods of the maximum priority index (MPI) and weighted penalty model (WPM) were examined in CAT and OMST while the normalized weighted absolute deviation heuristic (NWADH) was used to assemble MST forms. In addition to the complexity of the test specifications, the representation of each content category in the pool and on the test, size of the item pool, length of each stage, and number of preassembled MST difficulty levels were also varied. The performance of each test design was evaluated by three outcomes: content alignment, measurement precision, and test security. Results show that increasing the complexity of test specifications leads to worse content alignment across all test designs and item selection methods. The WPM item selection method performs better than the MPI and NWADH under increased constraint complexity. Moreover, CAT and OMST provide higher measurement precision than MST, especially for the large item pool. Finally, CAT is the most secure among the three test designs and the security of MST benefits most from the larger item pool.