Leakage and the reproducibility crisis in machine-learning-based science

🤖 Plain-English Summary

Machine-learning (ML) methods have gained prominence in the quantitative sciences. Finally, we conduct a reproducibility study of civil war prediction, where complex ML models are believed to vastly outperform traditional statistical models such as logistic regression (LR).

🔑 Key Findings

• However, there are many known methodological pitfalls, including data leakage, in ML-based science.
• We systematically investigate reproducibility issues in ML-based science.
• Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions.

🔗 Related Resources