4.6 Article

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

相关参考文献

注意:仅列出部分参考文献,下载原文获取全部文献信息。
Editorial Material Biology

Reimagining peer review as an expert elicitation process

Alexandru Marcoci et al.

Summary: Journal peer review plays an important role in regulating the flow of ideas in academic disciplines. However, research shows that editors cannot accurately identify the best experts for peer review. To prevent biases and uneven power distributions, introducing greater transparency and structure into the process is crucial.

BMC RESEARCH NOTES (2022)

Article Multidisciplinary Sciences

Mathematically aggregating experts' predictions of possible futures

A. M. Hanea et al.

Summary: Structured protocols provide a transparent and systematic way to aggregate probabilistic predictions from multiple experts. By using mathematical rules for aggregation, the objectivity and quality of predictions can be enhanced and measured through accuracy, calibration, and informativeness. Performance-based weighted aggregation can be effective when experts' performance can be scored beforehand, while other aggregation methods informed by measurable proxies for good performance can also be considered.

PLOS ONE (2021)

Article Economics

Expert forecasting with and without uncertainty quantification and weighting: What do the data say?

Roger M. Cooke et al.

Summary: Post-2006 expert judgment data have been extended to include 530 experts assessing 580 calibration variables from their fields. New analysis shows that point predictions as medians of combined expert distributions outperform combined medians, and medians of performance weighted combinations outperform medians of equally weighted combinations, resulting in a 65% improvement. The Random Expert Hypothesis underlying all performance-blind combination schemes is tested by randomly scrambling expert panels, rejecting the hypothesis that differences in expert performance reflect random stressors.

INTERNATIONAL JOURNAL OF FORECASTING (2021)

Review Biology

Journal policies and editors' opinions on peer review

Daniel G. Hamilton et al.

Article Psychology

Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully

Suzanne Hoogeveen et al.

ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE (2020)

Article Multidisciplinary Sciences

Predicting the replicability of social science lab experiments

Adam Altmejd et al.

PLOS ONE (2019)

Article Psychology, Multidisciplinary

When and Why to Replicate: As Easy as 1, 2, 3?

Sarahanne M. Field et al.

COLLABRA-PSYCHOLOGY (2019)

Article Public, Environmental & Occupational Health

The Value of Performance Weights and Discussion in Aggregated Expert Judgments

Anca M. Hanea et al.

RISK ANALYSIS (2018)

Article Psychology, Biological

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Colin F. Camerer et al.

NATURE HUMAN BEHAVIOUR (2018)

Article Ecology

A practical guide to structured expert elicitation using the IDEA protocol

Victoria Hemming et al.

METHODS IN ECOLOGY AND EVOLUTION (2018)

Article Biochemistry & Molecular Biology

Can cancer researchers accurately judge whether preclinical reports will reproduce?

Daniel Benjamin et al.

PLOS BIOLOGY (2017)

Article Multidisciplinary Sciences

Evaluating replicability of laboratory experiments in economics

Colin F. Camerer et al.

SCIENCE (2016)

Article Multidisciplinary Sciences

Using prediction markets to estimate the reproducibility of scientific research

Anna Dreber et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2015)

Article Multidisciplinary Sciences

Estimating the reproducibility of psychological science

Alexander A. Aarts et al.

SCIENCE (2015)

Editorial Material Multidisciplinary Sciences

Promoting an open research culture

B. A. Nosek et al.

SCIENCE (2015)

Article Psychology, Multidisciplinary

Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions

Barbara Mellers et al.

PERSPECTIVES ON PSYCHOLOGICAL SCIENCE (2015)

Article Psychology, Social

Investigating Variation in Replicability A Many Labs'' Replication Project

Richard A. Klein et al.

SOCIAL PSYCHOLOGY (2014)

Article Economics

Combining multiple probability predictions using a simple logit model

Ville A. Satopaeae et al.

INTERNATIONAL JOURNAL OF FORECASTING (2014)

Article Mathematics

Aggregating expert judgement

Simon French

REVISTA DE LA REAL ACADEMIA DE CIENCIAS EXACTAS FISICAS Y NATURALES SERIE A-MATEMATICAS (2011)

Article Public, Environmental & Occupational Health

Reducing Overconfidence in the Interval Judgments of Experts

Andrew Speirs-Bridge et al.

RISK ANALYSIS (2010)

Article Medicine, General & Internal

The relationship of previous training and experience of journal peer reviewers to subsequent review quality

Michael L. Callaham et al.

PLOS MEDICINE (2007)

Article Psychology, Experimental

When 90% confidence intervals are 50% certain: On the credibility of credible intervals

KH Teigen et al.

APPLIED COGNITIVE PSYCHOLOGY (2005)

Review Psychology

Group performance and decision making

NL Kerr et al.

ANNUAL REVIEW OF PSYCHOLOGY (2004)