ETS Reduces Costs With Questions Inventory Management

In the mid-1990s, ETS started a transition from paper-and-pencil testing to computer-based testing for all of its graduate-level tests.  Computer-based tests, offered on an almost daily basis at many test locations around the world and throughout the year, draw questions from a weekly pool of test items many times larger than the test itself, and that weekly pool itself gets refreshed from an even larger inventory of items.  This inventory represents a substantial investment to ETS, because the items must be written and honed for precision, reviewed for cultural and gender bias, and pre-tested on large samples of test takers. Both item security and millions of dollars in item development costs are affected by the way in which the item inventory is managed. 

An SD model was developed to project the number of available test items under different assumptions about security risk, and also to look at policies for more cost-effective inventory management.  The model revealed that in cases of higher security risk, under existing policies, the number of available items could stagnate at an unacceptably low level.  Model tests showed that a modest and safe amount of “recycling” items used in the past could neutralize this potential problem.  Moreover, the model showed that, even when security risk was not high, the policy of recycling could effectively reduce ETS item creation costs and help their bottom line.  The model analysis was presented to the ETS executive board and affected their decisions regarding test item inventory policy.

Do you want to know more?

Publication