Machine Learning and Artificial Intelligence Forecasting

There is a huge and growing amount of both observed data and forecast products.   Artificial Intelligence (AI) and Machine Learning (ML) are able to deal with massive amounts of data economically.  They bring major speed and cost benefits to the practice of weather forecasting.   AI is destined to play a substantial role in meteorological forecasting in the future. 

AI as applied to weather forecasting is radically different to the physics based techniques currently widely used.  There is no need to explicitly simulate the physics of the atmosphere;  this is currently implicit in the re-analysis training data used.  The user may not even know the impact of the local or large scale physical effects upon the process.

The major difference between forecasting techniques is:

  • traditional weather forecasting models assume a reasonably accurate physical model of the Earth system.  The biggest unknown is the initial conditions from which to start the forecast.
  • AI and ML do not explicitly handle the physics of the atmosphere.  The aim is to define an empirical model that links observational data at one time with similar data for a short time later.  Currently the empirical model has been developed (or trained) on reanalysis fields as observations and compared with reanalysis fields for the later time.  Having found this empirical model it can be used iteratively to compute forecast fields at successive time steps into the future.

Before an AI forecasting system can be implemented it has to have been trained on a large amount of observed data.  At ECMWF the AIFS is trained mainly to produce six hour forecasts.  

Machine Learning uses statistical methods and numerical optimisation to define and incrementally improve the relationship between a set of different observations and forecast values at a later time.   AIFS uses Graph Neural Networks which allows flexibility with grids and parameter efficiency.

Artificial Intelligence forecasting uses the algorithms derived by Machine Learning to produce forecasts.  These can be as a single forecast (AIFS Single) or as an ensemble of forecasts (AIFS-ENS).

Nevertheless, physics-based numerical weather prediction models remain key for these fully ML approaches. The IFS is used to create both training and validation data (using ERA5 and a few years of operational analysis).  AIFS and other machine learning models have been trained to minimise some measure of the error of forecast parameters.  In this way they have ordinarily been trained for use as a deterministic model.  In turn, AIFS relies on the IFS to provide initial conditions for each forecast.

Currently the main contrasts between the ECMWF models are:

  • IFS has high temporal, horizontal and vertical resolution and is unparalleled for the breadth of predicted variables.  ECMWF remains firmly committed to further improvement of the IFS.  
  • AIFS has lower temporal, horizontal and vertical resolution and has relatively limited number of predicted variables (Table1).  ML models are evolving quickly and the number of variables accounted for will increase as AIFS develops.

Ensemble forecasting (AIFS-ENS) brings an ability to quantify uncertainty and to identify possible extremes in the evolution.  An ensemble of the AI forecast procedure can be executed because AI forecasting is rapid and cheap to run.  At present AIFS-ENS has 50 members plus a nominal "control run" and is run at six hour intervals.   However there is potential for much larger ensembles and/or more frequent forecasts.

Multi-date verification suggests ML broadscale forecasts score better than classical NWP.   However, shorter wave length features and fine detail tend to not be well predicted in deterministic mode, particularly as forecast lead time increases. However, this is much less of an issue for AIFS-ENS.


Users should try to stay up to date with the latest AIFS developments.


Fig2.1.6-1: Difference of concept between traditional physics-based weather forecasting models and Artificial Intelligence models.  Physics-based models process observational data using physics equations as rules to provide forecasts of variables as output.  Machine Learning (or training) uses data from ERAS5 as input, checks the forecast results against later ERA5 data, and provides forecast algorithms. Once defined these algorithms are used by subsequent AI forecasts.  The AI model processes observational data using the algorithms to provide forecasts of variables as output.

Fig2.1.6-2: Comparing IFS and AI forecasting suites and flow and approximate size of data. 


Table2.1.6-1: Observed and forecast variables and constants at pressure levels used by the machine learning process and forecast process within ECMWF AIFS.  AIFS receives as input a representation of the state of the atmosphere - from ERA5 for the ML process; from the ECMWF operational analysis for the AI forecast process.  Both ML during the training process and AI during the forecast process predict the atmospheric state six hours into the future (with repetitions of this procedure in the forecast process enabling much longer lead time forecasts to be created)Currently AIFS only uses data at the surface and at standard pressure levels (diagram on the right).



Additional Sources of Information

(Note: In older material there may be references to issues that have subsequently been addressed)



(FUG Associated with Cy49r1)




 

  • No labels