An item sorting heuristic to derive equivalent parallel test versions from multivariate items

Parallel test versions require a comparable degree of difficulty and must capture the same characteristics using different items. This can become challenging when dealing with multivariate items, which are for example very common in language or image data. Here, we propose a heuristic to identify and select similar multivariate items for the generation of equivalent parallel test versions. This heuristic includes: 1. inspection of correlations between variables; 2. identification of outlying items; 3. application of a dimension-reduction method, such as for example principal component analysis (PCA); 4. generation of a biplot, in case of PCA of the first two principal components (PC), and grouping the displayed items; 5. assigning of the items to parallel test versions; and 6. checking the resulting test versions for multivariate equivalence, parallelism, reliability, and internal consistency. To illustrate the proposed heuristic, we applied it exemplarily on the items of a picture naming task. From a pool of 116 items, four parallel test versions were derived, each containing 20 items. We found that our heuristic can help to generate parallel test versions that meet requirements of the classical test theory, while simultaneously taking several variables into account.

Automated summary

Parallel test versions need to have a comparable difficulty and capture the same characteristics using different items. However, this becomes challenging when dealing with multivariate items, which are common in language or image data. To address this, a heuristic is proposed to identify and select similar multivariate items for generating equivalent parallel test versions. The heuristic involves examining variable correlations, identifying outliers, applying dimension-reduction methods, generating a biplot, assigning items to test versions, and checking for multivariate equivalence, parallelism, reliability, and consistency. The application of the heuristic on a picture naming task resulted in four parallel test versions, demonstrating its effectiveness in incorporating multiple variables.