4.4 Article

Moving experimental psychology online: How to obtain high quality data when we can't see our participants

相关参考文献

注意:仅列出部分参考文献,下载原文获取全部文献信息。
Article Psychology, Experimental

Explaining World-Wide Variation in Navigation Ability from Millions of People: Citizen Science Project Sea Hero Quest

Hugo J. Spiers et al.

Summary: Navigation ability varies widely across humans, with age and gender playing a role. The impact of age and gender on navigation ability varies across countries and is partly influenced by gender inequality. Upbringing in a city has a negative impact on navigation ability, and navigation performance can help identify individuals at higher genetic risk of Alzheimer's disease.

TOPICS IN COGNITIVE SCIENCE (2023)

Article Multidisciplinary Sciences

How games can make behavioural science better

Bria Long et al.

NATURE (2023)

Article Psychology, Experimental

Cognitive Control Across the Lifespan: Congruency Effects Reveal Divergent Developmental Trajectories

Christopher D. Erb et al.

Summary: The Simon, Stroop, and flanker tasks are commonly used to assess cognitive control, but it is unclear if they measure the same abilities in the same proportion. Two massive online studies were conducted, and only the flanker task showed an inverted U-shaped age-related change, while the Simon and Stroop tasks did not decline significantly with age. The congruency effects observed in each task were not strongly correlated. These findings have implications for understanding cognitive control and the suitability of these tasks.

JOURNAL OF EXPERIMENTAL PSYCHOLOGY-GENERAL (2023)

Article Neurosciences

What over 1,000,000 participants tell us about online research protocols

Johanna Tomczak et al.

Summary: With the increasing use of online research tools, we now have access to macro-level trends in research that were previously unattainable. By analyzing data from Gorilla Experiment Builder and Prolific, we discovered important insights about the current state of online research, which can be used by researchers for their own study planning and execution.

FRONTIERS IN HUMAN NEUROSCIENCE (2023)

Article Psychology, Mathematical

jsQuestPlus: A JavaScript implementation of the QUEST plus method for estimating psychometric function parameters in online experiments

Daiichiro Kuroki et al.

Summary: This article introduces jsQuestPlus, a JavaScript library based on the Bayesian adaptive psychometric methods QUEST and QUEST+. It provides integrations with online experimental tools and has shown good performance in terms of computation time, stimulus parameter determination, and update time. The estimation of psychometric parameters is accurate, making jsQuestPlus suitable for online psychophysical experiments.

BEHAVIOR RESEARCH METHODS (2023)

Article Psychology, Mathematical

OWLET: An automated, open-source method for infant gaze tracking using smartphone and webcam recordings

Denise M. Werchan et al.

Summary: Groundbreaking insights into the origins of the human mind can be obtained through studying eye movements of preverbal subjects. A new open-source methodology called OWLET is introduced, which automatically analyzes infant eye-tracking data collected on personal devices in the home. OWLET provides reliable estimation of infants' point-of-gaze and fills a significant gap in current tools for rapid online data collection and large-scale assessments of cognitive processes in infants.

BEHAVIOR RESEARCH METHODS (2023)

Article Psychology, Mathematical

Prevalence norms for 40,777 Catalan words: An online megastudy of vocabulary size

Marc Guasch et al.

Summary: This study provides word prevalence data for 40,777 Catalan words and examines the characteristics of both participants and words that influence word knowledge. The results show that age is the main factor influencing vocabulary size for participants, while lexical frequency is the most determining factor for word knowledge.

BEHAVIOR RESEARCH METHODS (2023)

Article Psychology, Mathematical

To BYOD or not: Are device latencies important for bring-your-own-device (BYOD) smartphone cognitive testing?

Jessica Nicosia et al.

Summary: Studies using remote cognitive testing need to decide whether to allow participants to use their own devices or provide study-specific devices. BYOD studies offer advantages but lack control over device performance, which could affect results. This study found significant variations in display and touch latencies among different smartphones, highlighting the need to account for these differences.

BEHAVIOR RESEARCH METHODS (2023)

Article Psychology, Mathematical

Precise display time measurement in JavaScript for web-based experiments

Gaspar Lukacs et al.

Summary: Conducting research online is a popular option for behavioral scientists, but the timing of display changes on web-browsers can be improved. This study found that using a recursive requestAnimationFrame (RAF) loop significantly improved precision compared to a single RAF call. Additionally, visibility or opacity changes were recommended for image stimuli. The study provides easy-to-use code for implementing this timing method.

BEHAVIOR RESEARCH METHODS (2023)

Article Psychology, Mathematical

The Headphone and Loudspeaker Test - Part I: Suggestions for controlling characteristics of playback devices in internet experiments

Yves Wycisk et al.

Summary: This study developed and evaluated a comprehensive, efficient, and easy-to-handle test procedure for remotely controlling and identifying playback device characteristics in online experiments. The procedure has shown high test-retest reliability and can help improve data quality in internet experiments.

BEHAVIOR RESEARCH METHODS (2023)

Article Psychology, Mathematical

Audio-Tokens: A toolbox for rating, sorting and comparing audio samples in the browser

Peter W. Donhauser et al.

Summary: A JavaScript toolbox is described for online rating studies with auditory material. The toolbox allows the collection of various types of ratings by associating audio samples with visual tokens. It can be used for psychological research on speech and music perception, as well as for dataset curation and annotation in machine learning.

BEHAVIOR RESEARCH METHODS (2023)

Review Psychology, Developmental

Developmental psychologists should adopt citizen science to improve generalization and reproducibility

Wei Li et al.

Summary: The widespread failures of replication and generalization in the field of developmental psychology confirm the metascientific theory and highlight the need for a mechanism to collect data on a larger scale and from more diverse populations. Citizen science, which involves uncompensated volunteers providing data, offers a potential solution to this problem. The article presents examples, discusses challenges, and compares citizen science to other methods of data collection.

INFANT AND CHILD DEVELOPMENT (2022)

Article Psychology

Journal N-Pact Factors From 2011 to 2019: Evaluating the Quality of Social/Personality Journals With Respect to Sample Size and Statistical Power

R. Chris Fraley et al.

Summary: Scholars and institutions commonly use impact factors to evaluate the quality of empirical research. However, a number of findings published in journals with high impact factors have failed to replicate, suggesting that impact alone may not be an accurate indicator of quality. Fraley and Vazire proposed an alternative index, the N-pact factor, which indexes the median sample size of published studies, providing a narrow but relevant indicator of research quality. In the present research, we expand on the original report by examining the N-pact factor of social/personality-psychology journals between 2011 and 2019, incorporating additional journals and accounting for study design (i.e., between persons, repeated measures, and mixed). There was substantial variation in the sample sizes used in studies published in different journals. Journals that emphasized personality processes and individual differences had larger N-pact factors than journals that emphasized social-psychological processes. Moreover, N-pact factors were largely independent of traditional markers of impact. Although the majority of journals in 2011 published studies that were not well powered to detect an effect of rho = .20, this situation had improved considerably by 2019. In 2019, eight of the nine journals we sampled published studies that were, on average, powered at 80% or higher to detect such an effect. After decades of unheeded warnings from methodologists about the dangers of small-sample designs, the field of social/personality psychology has begun to use larger samples. We hope the N-pact factor will be supplemented by other indices that can be used as alternatives to improve further the evaluation of research.

ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE (2022)

Article Psychology, Mathematical

Internet-based language production research with overt articulation: Proof of concept, challenges, and practical advice

Anne Vogt et al.

Summary: This study confirms the reliable online measurement method of key naming latency in language production research, and demonstrates the feasibility of using the picture-word interference paradigm for language production experiments in online settings.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Mathematical

Data quality of platforms and panels for online behavioral research

Peer Eyal et al.

Summary: The study found significant differences in data quality aspects among different platforms and panels in online behavioral research, especially in comprehension, attention, and honesty. Reputation could not predict data quality, but frequency and purpose of usage did, especially on MTurk.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Mathematical

Web-based language production experiments: Semantic interference assessment is robust for spoken and typed response modalities

Kirsten Stark et al.

Summary: For experimental research on language production, temporal precision and high quality of the recorded audio files are indispensable. Online research has the potential to improve efficiency, ecological validity, and diversity of study populations, and can yield similar results with reduced workload.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Mathematical

Does online masked priming pass the test? The effects of prime exposure duration on masked identity priming

Bernhard Angele et al.

Summary: We conducted online experiments to test the feasibility of using masked priming technique online, and found that the online data is similar to previous laboratory data. We also observed the effect of prime duration on the results, providing confidence in using masked priming online.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Mathematical

REPP: A robust cross-platform solution for online sensorimotor synchronization experiments

Manuel Anglada-Tort et al.

Summary: Sensorimotor synchronization (SMS), the rhythmic coordination of perception and action, is a fundamental human skill. This study presents REPP, a novel technology for measuring SMS in online experiments. REPP achieves high temporal accuracy, test-retest reliability, and concurrent validity. It is fully automated and customizable, allowing researchers to monitor experiments in real time and implement a wide variety of SMS paradigms.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Mathematical

The Online Coalition Game: A tool for online interactive coalition formation research

Joeri Wissink et al.

Summary: In this paper, we introduce the Online Coalition Game (OCG) as an open-source tool for conducting high-powered interactive coalition formation experiments on the oTree research platform. Through two previous implementations, we illustrate the advantages of using OCG online, including large sample sizes, fast data collection, and robust findings. Furthermore, we demonstrate that small changes in the experimental setup provide interesting opportunities to expand coalition formation theory.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Mathematical

MouseView.js: Reliable and valid attention tracking in web-based experiments using a cursor-directed aperture

Alexander L. Anwyl-Irvine et al.

Summary: MouseView.js is an alternative to eye tracking in web-based research, providing a reliable and valid instrument for attention-tracking. Inspired by the visual system, MouseView.js blurs the display to mimic peripheral vision while allowing participants to move a sharp aperture for fixating on stimuli of interest.

BEHAVIOR RESEARCH METHODS (2022)

Article Psychology, Multidisciplinary

A Global Perspective on Testing Infants Online: Introducing ManyBabies-AtHome

Lorijn Zaadnoordijk et al.

Summary: Online testing shows great promise for infant scientists by increasing diversity, improving reproducibility, and reducing costs. However, challenges remain in making this approach globally available, requiring international collaboration to address technical and ethical concerns.

FRONTIERS IN PSYCHOLOGY (2021)

Review Psychology, Multidisciplinary

Organizing the Methodological Toolbox: Lessons Learned From Implementing Developmental Methods Online

Jonathan F. Kominsky et al.

Summary: Adapting traditional lab, preschool, or museum studies to online data collection poses challenges, but specific solutions can be tailored based on the research questions and available technology constraints. This study discusses approaches for adapting studies targeting various developmental populations using different online methods during the COVID-19 pandemic, presenting solutions for recruitment and representativeness issues.

FRONTIERS IN PSYCHOLOGY (2021)

Article Psychology, Experimental

More evidence from over 1.1 million subjects that the critical period for syntax closes in late adolescence

Tony Chen et al.

Summary: The ability to achieve native-like proficiency in a second language is closely related to the age at which learning begins, with research showing that the ability to learn syntax sharply declines starting at the end of adolescence. The literature is still divided on the exact properties of this phenomenon.

COGNITION (2021)

Article Medicine, General & Internal

The pandemic of online research in times of COVID-19

Jeroen De Man et al.

Summary: The COVID-19 pandemic has led to a surge in online research using rating scales, but challenges such as selection bias and non-validated scales can affect the quality of this research. Researchers need to use validated scales and maintain a critical stance towards this methodology.

BMJ OPEN (2021)

Article Psychology, Multidisciplinary

Moderated Online Data-Collection for Developmental Research: Methods and Replications

Aaron Chuey et al.

Summary: Online data collection methods, especially through video conferencing software for face-to-face interaction, were discussed in this article with a focus on moderated data collection methods for young children. Comparative results showed that the outcomes of online and in-person methods are comparable, providing empirical support for the validity of online data collection methods.

FRONTIERS IN PSYCHOLOGY (2021)

Article Ophthalmology

Controlling the spatial dimensions of visual stimuli in online experiments

Jan W. Brascamp

Summary: This paper examines three approaches to estimating viewing distance online for human subjects' research, with the best method resulting in an average expected error close to 10% of the intended stimulus size.

JOURNAL OF VISION (2021)

Article Psychology, Mathematical

Cognitive test scores vary with choice of personal digital device

Eliza Passell et al.

Summary: Mobile- and web-based psychological research offer advantages in terms of recruitment, but also bring challenges related to differences in device latency which can affect cognitive performance on reaction time tests. Users of different devices show significant differences in measured performance on tests requiring fast reactions or fine motor movements.

BEHAVIOR RESEARCH METHODS (2021)

Article Psychology, Mathematical

Do data from mechanical Turk subjects replicate accuracy, response time, and diffusion modeling results?

Roger Ratcliff et al.

Summary: The study utilized online data collection to examine subject performance, finding that subjects performed relatively well in lexical decision and item recognition tasks, while a significant number of subjects in numerosity discrimination tasks exhibited fast guesses and unstable RTs.

BEHAVIOR RESEARCH METHODS (2021)

Article Psychology, Mathematical

An online headphone screening test based on dichotic pitch

Alice E. Milne et al.

Summary: Online experimental platforms can be an alternative or complement to lab-based research, but limited control over participants' listening environment is a concern. Headphones offer better sound control and noise shielding, and a new HP test shows promise for detecting headphone users with high selectivity. Combining tests can further lower false-positive rates, particularly beneficial for critical headphone use cases.

BEHAVIOR RESEARCH METHODS (2021)

Article Psychology, Mathematical

Realistic precision and accuracy of online experiment platforms, web browsers, and devices

Alexander Anwyl-Irvine et al.

Summary: Online behavioral research is booming due to its increasing ease of use and ability to quickly collect large samples. Researchers have conducted automated tests to understand how different web-building platforms, browsers, and operating systems impact experiment presentation and data quality. Findings show that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, with no single platform standing out as the best in all features and conditions. Participants' demographics, technology, and software usage were also analyzed to provide context to the research findings.

BEHAVIOR RESEARCH METHODS (2021)

Article Social Sciences, Interdisciplinary

Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks

Andie Storozuk et al.

QUANTITATIVE METHODS FOR PSYCHOLOGY (2020)

Article Psychology, Mathematical

Mental chronometry in the pocket? Timing accuracy of web applications on touchscreen and keyboard devices

Thomas Pronk et al.

BEHAVIOR RESEARCH METHODS (2020)

Article Psychology, Mathematical

Gorilla in our midst: An online behavioral experiment builder

Alexander L. Anwyl-Irvine et al.

BEHAVIOR RESEARCH METHODS (2020)

Article Multidisciplinary Sciences

Toward personalized cognitive diagnostics of at-genetic-risk Alzheimer's disease

Gillian Coughlan et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2019)

Editorial Material Behavioral Sciences

Preregistration Is Hard, And Worthwhile

Brian A. Nosek et al.

TRENDS IN COGNITIVE SCIENCES (2019)

Article Psychology

Research in Social Psychology Changed Between 2011 and 2016: Larger Sample Sizes, More Self-Report Measures, and More Online Studies

Kai Sassenberg et al.

ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE (2019)

Article Psychology, Mathematical

A thousand studies for the price of one: Accelerating psychological science with Pushkin

Joshua K. Hartshorne et al.

BEHAVIOR RESEARCH METHODS (2019)

Article Communication

RESEARCH SYNTHESIS SATISFICING IN SURVEYS: A SYSTEMATIC REVIEW OF THE LITERATURE

Caroline Roberts et al.

PUBLIC OPINION QUARTERLY (2019)

Article Multidisciplinary Sciences

The preregistration revolution

Brian A. Nosek et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2018)

Article Multidisciplinary Sciences

Toward a psychology of Homo sapiens: Making psychological science more representative of the human population

Mostafa Salari Rad et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2018)

Proceedings Paper Computer Science, Artificial Intelligence

Demographics and Dynamics of Mechanical Turk Workers

Djellel Difallah et al.

WSDM'18: PROCEEDINGS OF THE ELEVENTH ACM INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING (2018)

Article Business, Finance

Prolific.ac-A subject pool for online experiments

Stefan Palan et al.

JOURNAL OF BEHAVIORAL AND EXPERIMENTAL FINANCE (2018)

Article Psychology, Experimental

A critical period for second language acquisition: Evidence from 2/3 million English speakers

Joshua K. Hartshorne et al.

COGNITION (2018)

Proceedings Paper Computer Science, Hardware & Architecture

Volunteer-Based Online Studies With Older Adults and People with Disabilities

Qisheng Li et al.

ASSETS'18: PROCEEDINGS OF THE 20TH INTERNATIONAL ACM SIGACCESS CONFERENCE ON COMPUTERS AND ACCESSIBILITY (2018)

Review Behavioral Sciences

Crowdsourcing Samples in Cognitive Science

Neil Stewart et al.

TRENDS IN COGNITIVE SCIENCES (2017)

Article Psychology, Biological

A manifesto for reproducible science

Marcus R. Munafo et al.

NATURE HUMAN BEHAVIOUR (2017)

Article Psychology, Social

Beyond the Turk: Alternative platforms for crowdsourcing behavioral research

Eyal Peer et al.

JOURNAL OF EXPERIMENTAL SOCIAL PSYCHOLOGY (2017)

Article Psychology, Mathematical

TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences

Leib Litman et al.

BEHAVIOR RESEARCH METHODS (2017)

Review Psychology, Clinical

Conducting Clinical Research Using Crowdsourced Convenience Samples

Jesse Chandler et al.

ANNUAL REVIEW OF CLINICAL PSYCHOLOGY, VOL 12 (2016)

Article Psychology, Mathematical

Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants

David J. Hauser et al.

BEHAVIOR RESEARCH METHODS (2016)

Article Psychology, Mathematical

Online versus offline: The Web as a medium for response time data collection

Andrey Chetverikov et al.

BEHAVIOR RESEARCH METHODS (2016)

Article Psychology, Social

The Pitfall of Experimenting on the Web: How Unattended Selective Attrition Leads to Surprising (Yet False) Research Conclusions

Haotian Zhou et al.

JOURNAL OF PERSONALITY AND SOCIAL PSYCHOLOGY (2016)

Review Psychology

Internet Research in Psychology

Samuel D. Gosling et al.

ANNUAL REVIEW OF PSYCHOLOGY, VOL 66 (2015)

Article Ethics

Detecting, Preventing, and Responding to Fraudsters in Internet Research: Ethics and Tradeoffs

Jennifer E. F. Teitcher et al.

JOURNAL OF LAW MEDICINE & ETHICS (2015)

Article Multidisciplinary Sciences

Estimating the reproducibility of psychological science

Alexander A. Aarts et al.

SCIENCE (2015)

Article Multidisciplinary Sciences

Collecting Psycholinguistic Response Time Data Using Amazon Mechanical Turk

Kelly Enochson et al.

PLOS ONE (2015)

Proceedings Paper Computer Science, Theory & Methods

LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples

Katharina Reinecke et al.

PROCEEDINGS OF THE 2015 ACM INTERNATIONAL CONFERENCE ON COMPUTER-SUPPORTED COOPERATIVE WORK AND SOCIAL COMPUTING (CSCW'15) (2015)

Article Psychology, Mathematical

QRTEngine: An easy solution for running online reaction time experiments using Qualtrics

Jonathan S. Barnhoorn et al.

BEHAVIOR RESEARCH METHODS (2015)

Article Psychology, Multidisciplinary

Using Nonnaive Participants Can Reduce Effect Sizes

Jesse Chandler et al.

PSYCHOLOGICAL SCIENCE (2015)

Article Psychology, Multidisciplinary

When Does Cognitive Functioning Peak? The Asynchronous Rise and Fall of Different Cognitive Abilities Across the Life Span

Joshua K. Hartshorne et al.

PSYCHOLOGICAL SCIENCE (2015)

Article Social Sciences, Interdisciplinary

It's a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on Tricky Tasks

David J. Hauser et al.

SAGE OPEN (2015)

Article Psychology, Mathematical

jsPsych: A JavaScript library for creating behavioral experiments in a Web browser

Joshua R. de Leeuw

BEHAVIOR RESEARCH METHODS (2015)

Article Psychology, Mathematical

Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Eyal Peer et al.

BEHAVIOR RESEARCH METHODS (2014)

Article Psychology, Multidisciplinary

Inside the Turk: Understanding Mechanical Turk as a Participant Pool

Gabriele Paolacci et al.

CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE (2014)

Article Multidisciplinary Sciences

Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

Matthew J. C. Crump et al.

PLOS ONE (2013)

Article Multidisciplinary Sciences

Number sense across the lifespan as revealed by a massive Internet-based sample

Justin Halberda et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2012)

Article Psychology, Mathematical

Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments

Laura Germine et al.

PSYCHONOMIC BULLETIN & REVIEW (2012)

Article Political Science

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

Adam J. Berinsky et al.

POLITICAL ANALYSIS (2012)

Article Communication

The Future of Modes of Data Collection

Mick P. Couper

PUBLIC OPINION QUARTERLY (2011)

Article Psychology, Biological

The weirdest people in the world?

Joseph Henrich et al.

BEHAVIORAL AND BRAIN SCIENCES (2010)

Review Psychology

Missing Data Analysis: Making It Work in the Real World

John W. Graham

ANNUAL REVIEW OF PSYCHOLOGY (2009)

Review Psychology

Mechanisms of Masked Priming: A Meta-Analysis

Eva Van den Bussche et al.

PSYCHOLOGICAL BULLETIN (2009)

Article Psychology, Social

Instructional manipulation checks: Detecting satisficing to increase statistical power

Daniel M. Oppenheimer et al.

JOURNAL OF EXPERIMENTAL SOCIAL PSYCHOLOGY (2009)