Page History

Machine Learning

The aim of Machine Learning (ML) is to continually develop (or train) an empirical model directly from observations or reanalyses.  Observations implicitly contain the physics of the atmosphere but it is not necessary for ML models emulate the underpinning physics that dictates the evolution of variables through a forecast.   During the ML training process, ML considers all the set of observed or initial data, and using statistical methods relates these to observed variable (e.g.temperature) six hours later at each point.  The initial data and corresponding data at the end of the forecast period have been extracted from some 20 years of ERA5 data.  At ECMWF, machine learning training is aimed towards producing six hour forecasts.  Table1 gives the set of observed and forecast variables and the constants considered during the machine learning process at ECMWF.

General Process

At each grid point the set of observed data is processed using the set of random weighting functions for each parameter.  Initially the forecast value will not agree with those observed at the verifying time of the forecast.  The error (loss function) as measured by some error metric is fed backwards (back propagation) within the process.  In response, the influence of types of observations (say wind, 50hPa temperature, etc.) may be reduced while that of others (say surface temperature) may be increased.  This process is repeated many times with the aim to progressively minimise the error metric. 

...

The Machine Learning process derives algorithms for AIFS Single and AIFS ENS in slightly different ways.

Considerations

The ML training process uses a vast amount of data and is very expensive in computer time and energy.  However, the process is executed only once to develop the observation-to-forecast relationships used subsequently by AIFS.    At rare intervals ML may be repeated to include further data types or different techniques etc.  In effect ML uses previous evolutions of similar analyses at several levels to “learn” the more likely evolution.   Although the ML model has no need to incorporate physical processes during the learning process it does provide some insights and seems to be learning some physics. 

...