4.7 Article

The importance of being external. methodological insights for the external validation of machine learning models in medicine

Related references

Note: Only part of the references are listed.
Article Medical Laboratory Technology

Development, evaluation, and validation of machine learning models for COVID-19 detection based on routine blood tests

Federico Cabitza et al.

Summary: The study found that machine learning models can identify COVID-19 positive patients quickly and economically using blood test data. ML can serve as an alternative method to rRT-PCR, especially beneficial for resource-constrained countries.

CLINICAL CHEMISTRY AND LABORATORY MEDICINE (2021)

Article Mathematical & Computational Biology

The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

Davide Chicco et al.

Summary: In this manuscript, the authors reaffirm the robustness of the Matthews correlation coefficient (MCC) as a metric for summarizing classifier performance. They compare MCC to other metrics such as balanced accuracy, bookmaker informedness, and markedness, and demonstrate situations where MCC provides a more informative response. The authors also identify three exceptions where bookmaker informedness may be more appropriate, but overall suggest MCC as a standard measure for evaluating binary classifications in various fields.

BIODATA MINING (2021)

Article Health Care Sciences & Services

Estimation of required sample size for external validation of risk models for binary outcomes

Menelaos Pavlou et al.

Summary: This paper investigates the sample size requirements for validation studies with binary outcomes to estimate measures of predictive performance, providing various estimators which perform well even when normality assumptions are violated. Our estimators show good performance, even when normality assumptions are violated.

STATISTICAL METHODS IN MEDICAL RESEARCH (2021)

Article Computer Science, Information Systems

The need to separate the wheat from the chaff in medical informatics Introducing a comprehensive checklist for the (self)-assessment of medical AI studies

Federico Cabitza et al.

Summary: The editorial proposes a practical checklist to help authors self-assess the quality of their contributions and aid reviewers in distinguishing high-quality medical ML studies from the mere application of ML techniques to medical data.

INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS (2021)

Article Health Care Sciences & Services

External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

Kym I. E. Snell et al.

Summary: Rules-of-thumb for sample size in external validation of clinical prediction models may not be precise, with factors like LP distribution affecting precision of performance estimates. A tailored simulation-based approach can offer more flexibility and reliability in determining sample size requirements for validation.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2021)

Article Mathematical & Computational Biology

Minimum sample size for external validation of a clinical prediction model with a binary outcome

Richard D. Riley et al.

Summary: External validation is crucial in examining the performance of prediction models, but current studies often face issues with small sample sizes. To address this, determining the minimum sample size needed for a new external validation study with precise estimation calculations is proposed, taking into account calibration, discrimination, and clinical utility measures.

STATISTICS IN MEDICINE (2021)

Article Computer Science, Artificial Intelligence

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

Michael Roberts et al.

Summary: Many machine learning-based approaches have been developed for the prognosis and diagnosis of COVID-19 from medical images. However, a systematic review found that current studies have methodological flaws, preventing their potential clinical utility. Recommendations are provided to address these issues for higher-quality model development.

NATURE MACHINE INTELLIGENCE (2021)

Article Mathematical & Computational Biology

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Lucinda Archer et al.

Summary: Clinical prediction models offer personalized outcome predictions for patient counseling and decision making, with external validation crucial for assessing model performance. Proposed criteria aim to determine minimum sample size needed for external validation of a clinical prediction model, considering factors like proportion of variance explained and agreement between predicted and observed values. The recommendations provide a framework for estimating precision and ensuring adequate sample sizes in future validation studies.

STATISTICS IN MEDICINE (2021)

Review Medical Laboratory Technology

Systematic review and meta-analysis of within-subject and between-subject biological variation estimates of 20 haematological parameters

Abdurrahman Coskun et al.

CLINICAL CHEMISTRY AND LABORATORY MEDICINE (2020)

Editorial Material Medicine, General & Internal

Challenges to the Reproducibility of Machine Learning Models in Health Care

Andrew L. Beam et al.

JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION (2020)

Article Medical Laboratory Technology

Routine blood tests as a potential diagnostic tool for COVID-19

Davide Ferrari et al.

CLINICAL CHEMISTRY AND LABORATORY MEDICINE (2020)

Article Biology

Automated detection of COVID-19 cases using deep neural networks with X-ray images

Tulin Ozturk et al.

COMPUTERS IN BIOLOGY AND MEDICINE (2020)

Article Computer Science, Information Systems

MINIMAR (MINimum Information for Medical Al Reporting): Developing reporting standards for artificial intelligence in health care

Tina Hernandez-Boussard et al.

JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION (2020)

Article Medical Informatics

As if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI

Federico Cabitza et al.

BMC MEDICAL INFORMATICS AND DECISION MAKING (2020)

Article Medical Laboratory Technology

Routine Laboratory Blood Tests Predict SARS-CoV-2 Infection Using Machine Learning

He S. Yang et al.

CLINICAL CHEMISTRY (2020)

Article Computer Science, Interdisciplinary Applications

COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios

Rodolfo M. Pereira et al.

COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE (2020)

Review Medicine, General & Internal

The optimal diagnostic methods for COVID-19

Tinotenda A. Harahwa et al.

DIAGNOSIS (2020)

Article Computer Science, Interdisciplinary Applications

CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images

Asif Iqbal Khan et al.

COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE (2020)

Article Computer Science, Interdisciplinary Applications

Explainable Deep Learning for Pulmonary Disease and Coronavirus COVID-19 Detection from X-rays

Luca Brunese et al.

COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE (2020)

Editorial Material Oncology

The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence

Federico Cabitza et al.

ANNALS OF TRANSLATIONAL MEDICINE (2019)

Article Multidisciplinary Sciences

Machine learning algorithm validation with a limited sample size

Andrius Vabalas et al.

PLOS ONE (2019)

Article Physics, Multidisciplinary

Using the Data Agreement Criterion to Rank Experts' Beliefs

Duco Veen et al.

ENTROPY (2018)

Article Mathematical & Computational Biology

Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

Gary S. Collins et al.

STATISTICS IN MEDICINE (2016)

Editorial Material Health Care Sciences & Services

Prediction models need appropriate internal, internal-external, and external validation

Ewout W. Steyerberg et al.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2016)

Article Health Care Sciences & Services

A calibration hierarchy for risk models was defined: from utopia to empirical data

Ben Van Calster et al.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2016)

Editorial Material Biochemical Research Methods

POINTS OF SIGNIFICANCE Model selection and overfitting

Jake Lever et al.

NATURE METHODS (2016)

Review Health Care Sciences & Services

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

Gary S. Collins et al.

BMC MEDICAL RESEARCH METHODOLOGY (2014)

Article Computer Science, Artificial Intelligence

On the effectiveness of preprocessing methods when dealing with different levels of class imbalance

V. Garcia et al.

KNOWLEDGE-BASED SYSTEMS (2012)

Article Medical Informatics

Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies

Valentin Rousson et al.

BMC MEDICAL INFORMATICS AND DECISION MAKING (2011)

Article Psychology, Multidisciplinary

Explaining physicians' acceptance of EHCR systems: An extension of TAM with trust and risk factors

Jose Manuel Ortega Egea et al.

COMPUTERS IN HUMAN BEHAVIOR (2011)

Article Hematology

Race-specific WBC and neutrophil count reference intervals

E. -M. Lim et al.

INTERNATIONAL JOURNAL OF LABORATORY HEMATOLOGY (2010)

Article Oncology

Receiver Operating Characteristic Curve in Diagnostic Test Assessment

Jayawant N. Mandrekar

JOURNAL OF THORACIC ONCOLOGY (2010)

Article Computer Science, Interdisciplinary Applications

Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap

Ji-Hyun Kim

COMPUTATIONAL STATISTICS & DATA ANALYSIS (2009)

Article Statistics & Probability

Diagnostics of prior-data agreement in applied Bayesian analysis

Nicolas Bousquet

JOURNAL OF APPLIED STATISTICS (2008)

Article Meteorology & Atmospheric Sciences

Sampling Uncertainty and Confidence Intervals for the Brier Score and Brier Skill Score

A. Allen Bradley et al.

WEATHER AND FORECASTING (2008)

Article Mathematical & Computational Biology

Practical experiences on the necessity of external validation

I. R. Koenig et al.

STATISTICS IN MEDICINE (2007)

Article Health Care Sciences & Services

Substantial effective sample sizes were required for external validation studies of predictive logistic regression models

Y Vergouwe et al.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2005)

Article Health Care Sciences & Services

External validation is necessary in, prediction research: A clinical example

SE Bleeker et al.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2003)

Article Health Care Sciences & Services

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

EW Steyerberg et al.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2003)

Article Health Care Sciences & Services

Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis

EW Steyerberg et al.

JOURNAL OF CLINICAL EPIDEMIOLOGY (2001)