🤖 AI System Predicts Natural Disasters with Up to 90% Accuracy
A "90% accurate" disaster AI claim says little without lead time and baselines. Where machine learning helps (floods, weather), where it does not (earthquakes), and how to check.
Headlines that an AI system predicts earthquakes, floods and hurricanes "with up to 90% accuracy" combine three very different hazards under one number. We could not tie the 90 percent figure in the original coverage to a named, peer-reviewed benchmark, so it should be treated as unverified. The more useful exercise for researchers is to ask what a figure like that could mean, which hazards machine learning has actually improved, and which it has not. The answers differ sharply by hazard, and the differences matter for anyone building or relying on warning systems.
Why a single accuracy number says little
Disasters are rare events, and accuracy is a poor metric for rare events. Suppose a damaging flood occurs on 1 percent of days at a gauge. A system that always predicts "no flood" is 99 percent accurate and completely useless. Forecast verification therefore uses measures that separate the kinds of error: probability of detection (the share of real events that were warned), false alarm ratio (the share of warnings that were wrong), and skill scores that compare a forecast with a reference such as climatology or persistence. Murphy's classic paper on forecast quality (Weather and Forecasting, 1993) argues that "goodness" has several independent dimensions, including consistency, quality and value to the user. Any claim should state the event definition, the lead time, the verification period, and the baseline.
Lead time is equally important. A system that "predicts" a cyclone one hour before landfall adds little. One that issues a reliable flood warning two days ahead changes what communities can do. Accuracy that is quoted without lead time cannot be compared with anything.
Earthquakes: warning is not prediction
Deterministic earthquake prediction, specifying time, place and magnitude in advance, has not been achieved, and decades of proposed precursors have not held up under scrutiny (Geller, Geophysical Journal International, 1997). What does work operationally is early warning: once a fault starts rupturing, the fast P-waves are detected near the epicentre and an alert can reach distant places seconds before the damaging S-waves and surface waves arrive. Probabilistic forecasting, meaning the changing likelihood of shaking over weeks to years, is the other established tool (Jordan and colleagues, Annals of Geophysics, 2011).
Machine learning has produced interesting results at the edges of this problem. In laboratory experiments with sheared fault material, models trained on continuous acoustic signals could estimate time to failure (Rouet-Leduc and colleagues, Geophysical Research Letters, 2017). Whether this transfers to the heterogeneous, sparsely instrumented real Earth is an open question. A deep-learning model of aftershock locations reported good performance (DeVries and colleagues, Nature, 2018), but a later analysis showed that a much simpler model with a single neuron performed comparably, which suggests that part of the apparent gain came from physical features that simple statistics could already use (Mignan and Broccardo, Nature, 2019). That episode is a good template for how to read a high-accuracy claim: ask what the simplest competitive baseline is.
Floods: where the evidence is strongest
Flood forecasting is the hazard where machine learning has the most convincing record, partly because river discharge is a continuous, well-measured quantity with long records. Kratzert and colleagues (Hydrology and Earth System Sciences, 2019) showed that a recurrent neural network trained across many catchments could predict streamflow in basins without local calibration, outperforming a standard hydrological model on several measures. Nearing and colleagues (Nature, 2024) extended this to a global model that gave reliable forecasts of extreme flood events in many places where no stream gauge exists, with performance comparable at lead times of several days to what is now available in better-gauged regions. Even here, performance is lower for the rarest extremes, which are the events that matter most and are least represented in training data.
Tropical cyclones and weather
Data-driven global weather models such as GraphCast (Lam and colleagues, Science, 2023) and Pangu-Weather (Bi and colleagues, Nature, 2023) have matched or exceeded the skill of leading physics-based systems on many medium-range measures, including cyclone track, at a small fraction of the computing cost. Track forecasts were already good. The harder problems are intensity and rapid intensification, which depend on small-scale processes in the storm core and on the ocean beneath. Data-driven models trained on coarse reanalysis data tend to smooth extremes, so they can underestimate peak intensity. This is an area of active improvement, not a solved problem.
Limitations and open questions
- Distribution shift. A model trained on historical records is only as good as the assumption that the future resembles the past. Climate change alters the statistics of floods, heat and storms.
- Data gaps. Reliable training data are scarcest where vulnerability is high. Many river basins in the developing world have short or missing gauge records, and seismic networks are sparse in many regions.
- False alarms. Too many erode public trust and carry real costs, such as unnecessary evacuations. The acceptable false alarm ratio is a social and economic question as well as a statistical one.
- Interpretability. Forecasters and emergency managers need to know why a warning was issued. Black-box scores are hard to defend in public.
- Last-mile delivery. A correct forecast that does not reach households in time saves no one. Many successful warning systems, in cyclone-prone coastal areas for example, rely heavily on community networks and clear messaging rather than only on model skill.
Questions worth testing
- Is the headline metric reported against a simple baseline such as climatology, persistence or a one-parameter statistical model?
- How does performance change for the largest events, held out from training?
- Are the test periods and regions independent of the training data, or is there leakage across neighbouring basins and time windows?
- Are probabilistic forecasts calibrated, so that "70 percent" verifies about 70 percent of the time?
- What is the benefit in lives or losses at a given false alarm rate, in a cost-loss framework?
Machine learning is already improving some hazard forecasts, flood and medium-range weather above all. For earthquakes, the honest position is that early warning and probabilistic forecasting are real, and precise prediction is not. A single "90 percent" figure covering all hazards should be read as a sign that the details are missing.
References and further reading
- Murphy, A. H. What is a good forecast? An essay on the nature of goodness in weather forecasting. Weather and Forecasting, 1993.
- Geller, R. J. Earthquake prediction: a critical review. Geophysical Journal International, 1997.
- Jordan, T. H. et al. Operational earthquake forecasting: state of knowledge and guidelines for utilization. Annals of Geophysics, 2011.
- Rouet-Leduc, B. et al. Machine learning predicts laboratory earthquakes. Geophysical Research Letters, 2017.
- DeVries, P. M. R. et al. Deep learning of aftershock patterns following large earthquakes. Nature, 2018; and Mignan, A. and Broccardo, M. One neuron versus deep learning in aftershock prediction. Nature, 2019.
- Kratzert, F. et al. Toward improved predictions in ungauged basins: exploiting the power of machine learning. Water Resources Research, 2019.
- Nearing, G. et al. Global prediction of extreme floods in ungauged watersheds. Nature, 2024.
- Lam, R. et al. Learning skillful medium-range global weather forecasting. Science, 2023.
- Bi, K. et al. Accurate medium-range global weather forecasting with 3D neural networks. Nature, 2023.