Pruning inferior systems using subjective constraints with sequentially added thresholds

Abstract

We consider the problem of pruning inferior systems among a finite number of simulated systems using constraints that are stochastic (in that their performance measures need to be estimated through observations) and subjective (in that their thresholds can be tightened or relaxed). With subjective constraints, the decision maker can test multiple threshold values to determine how a set of feasible systems changes as constraints become stricter and use this information to prune systems or identify the system with the best performance. When the number of possible thresholds is large, the decision maker may want to start by obtaining the feasibility decisions with respect to a smaller subset of thresholds. Depending on the results, she can then add tighter or relaxed thresholds if many or no feasible systems have been identified. In this article, we present a multipass pruning (MPP) procedure that starts with a smaller set of thresholds in a first pass and adds more thresholds sequentially in later passes with the goal of pruning inferior systems efficiently. We prove the statistical validity of the proposed procedure and numerically demonstrate its efficiency in terms of the required number of observations for pruning inferior systems.