How Many Participants? How Many Trials? Maximizing the Power of Reaction Time Studies

Due to limitations in the resources available for carrying out reaction time (RT) experiments, researchers often have to choose between testing relatively few participants with relatively many trials each or testing relatively many participants with relatively few trials each. To compare the experimental power that would be obtained under each of these options, I simulated virtual experiments using subsets of participants and trials from eight large real RT datasets examining 19 experimental effects. The simulations compared designs using the first N-T trials from N-P randomly selected participants, holding constant the total number of trials across all participants, N-P x N-T. The [N-P, N-T] combination maximizing the power to detect each effect depended on how the mean and variability of that effect changed with practice. For most effects, power was greater in designs having many participants with few trials each rather than the reverse, suggesting that researchers should usually try to recruit large numbers of participants for short experimental sessions. In some cases, power for a fixed total number of trials across all participants was maximized by having as few as two trials per participant in each condition. Where researchers can make plausible predictions about how their effects will change over the course of a session, they can use those predictions to increase their experimental power.

Automated summary

Due to limitations in resources, researchers often need to choose between testing few participants with many trials or many participants with few trials in reaction time experiments. This study simulated virtual experiments using subsets of participants and trials from real datasets, finding that designs with many participants and few trials each generally had greater power to detect experimental effects. However, in some cases, as few as two trials per participant in each condition could maximize power. Researchers can increase experimental power by making plausible predictions about how their effects will change over the course of a session.