Sample Size Calculation and Optimal Design for Regression-Based Norming of Tests and Questionnaires

To prevent mistakes in psychological assessment, the precision of test norms is important. This can be achieved by drawing a large normative sample and using regression-based norming. Based on that norming method, a procedure for sample size planning to make inference on Z-scores and percentile rank scores is proposed. Sampling variance formulas for these norm statistics are derived and used to obtain the optimal design, that is, the optimal predictor distribution, for the normative sample, thereby maximizing precision of estimation. This is done under five regression models with a quantitative and a categorical predictor, differing in whether they allow for interaction and nonlinearity. Efficient robust designs are given in case of uncertainty about the regression model. Furthermore, formulas are provided to compute the normative sample size such that individuals' positions relative to the derived norms can be assessed with prespecified power and precision. Translational Abstract Normative studies are needed to derive reference values (or norms) for tests and questionnaires, so that psychologists can use them to assess individuals. Specifically, norms allow psychologists to interpret individuals' score on a test by comparing it with the scores of their peers (e.g., individuals with the same sex, age, and educational level) in the reference population. Because norms are also used to make decisions on individuals, such as the assignment to clinical treatment or remedial teaching, it is important that norms are precise (i.e., not strongly affected by sampling error in the sample on which the norms are based). This article shows how this goal can be attained in three steps. First, norms are derived using the regression-based approach, which is more efficient than the traditional approach of splitting the sample into subgroups based on demographic factors and deriving norms per subgroup. Specifically, the regression-based approach allows researchers to identify the predictors (e.g., demographic factors) that affect the test score of interest, and to use the whole sample to derive norms. Second, the design of the normative study (e.g., which age groups to include) is chosen such that the precision of the norms is maximized for a given total sample size for norming. Third, this total sample size is computed such that a prespecified power and precision are obtained.

Automated summary

To prevent errors in psychological assessment, it is important to have precise test norms. This study proposes a method for sample size planning based on regression-based norming to improve the precision of inference on Z-scores and percentile rank scores. The study derives sampling variance formulas and uses them to determine the optimal design for the normative sample. It also provides formulas to compute the normative sample size based on desired power and precision.