Page History

Overview

There are two steps required in an AI forecast system:

• First phase; supply a set of algorithms that relate initial data with predicted data for a certain time in the future.  This is accomplished using Machine Learning (ML) and is done once only with the introduction of each new Cycle and associated reanalysis data. 
• Second phase; use these algorithms to predict data for a certain time in the future using observed and analysed data.  This is accomplished using Artificial Intelligence (AI) forecasting and can be done many times or as frequently as a forecast is required.

Machine Learning (ML)

The aim of Machine Learning (ML) is to develop (or train) an empirical model directly from observations or reanalyses.  Observations implicitly contain the physics of the atmosphere but it is not necessary for ML models emulate the underpinning physics that dictates the evolution of variables through a forecast.   During the ML training process, ML considers all the set of observed or initial data, and using statistical methods relates these to observed variable (e.g.temperature) six hours later at each point.  The initial data and corresponding data at the end of the forecast period have been extracted from some 20 years of ERA5 data.  At ECMWF, machine learning training is aimed towards producing six hour forecasts.  Table2.2.1 gives the set of observed and forecast variables and the constants considered during the machine learning process at ECMWF.

Machine Learning Process

At each grid point the set of observed data is processed using the set of random weighting functions for each parameter.  Initially the forecast value will not agree with those observed at the verifying time of the forecast.  The error (loss function) as measured by some error metric is fed backwards (back propagation) within the process.  In response, the influence of types of observations (say wind, 50hPa temperature, etc.) may be reduced while that of others (say surface temperature) may be increased.  This process is repeated many times with the aim to progressively minimise the error metric (See Fig2B.1-1). 

...

Sometimes the ML model requires fine-tuning. This process doesn't require a full retraining of the model.  Instead, targeted adjustments to the model's weights and parameters reflect the new data and scientific findings. This selective updating helps ensure that the new information is not drowned out by the volume of pre-existing training data and avoids conflicts with established reanalyses. This keeps ECMWF ML models at the cutting edge.

Artificial Intelligence (AI) forecasting

The aim of AI forecasting is to use the algorithms developed in the Machine Learning phase to predict values of several parameters

The AI forecasting process

Essentially, at each grid point the AI forecasting process applies algorithms to forecast each variable.  These algorithms relate input data to forecast data and have been derived by machine learning (ML) training.   The forecasting process uses the complete set of available observed variables and produces a complete set of forecast variables for six hours later.  Two AI forecast systems are used at ECMWF:

...

All parameters are forecast individually.  So AI models do not necessarily, but normally do, produce physically and dynamically consistent predictions that are sufficiently skilful for all relevant scales.  Forecast wind may not exactly correspond to the forecast height or pressure gradient.

Image RemovedImage Added

Fig2B.1-1: Forecasting process using AI for a single parameter for a single step.  The algorithm to produce each single parameter uses all the set of input variables.  The algorithms relating the observed data to predicted value of each parameter six hours later have been derived by ML.  Note: In the diagram "other parameters" include 6hr precipitation and 6hr convective precipitation.

Image RemovedImage Added

Fig2B.1-2: Sequence of forecasting processes using AI for all the parameters for a complete 360hr (15day) forecast.   Each algorithm to produce each output variable uses all input variables.  The algorithms relating observed data to predicted data six hours later have been derived by ML. Note: In the diagram "other parameters" include 6hr precipitation and 6hr convective precipitation.

Image RemovedImage RemovedImage AddedImage Added

Table2B.1:  The ML machine learning process and the AIFS forecast process use observed and forecast variables and constants.  Currently AIFS only uses data at the surface and at standard pressure levels (diagram on the right).  Representation of the state of the atmosphere is taken from:  

...

Both ML during the training process and AI during the forecast process predict the atmospheric state for six hours in the future.  

Highlights and challenges

Strengths of using AI are:

...