Note: HRES and Ensemble Control Forecast (ex-HRES) are scientifically, structurally and computationally identical. With effect from Cy49r1, Ensemble Control Forecast (ex-HRES) output is equivalent to HRES output shown in the diagrams. At the time of the diagrams, HRES had resolution of 9km and ensemble members had a resolution of 18km.
The Anomaly Correlation Coefficient (ACC) is one of the most widely used measures in the verification of spatial fields. It is the spatial correlation between a forecast anomaly relative to climatology, and a verifying analysis anomaly relative to climatology. ACC is a measure of how well the forecast anomalies have represented the observed anomalies. It shows how well the predicted values from a forecast model "fit" with the real-life data. ACC for a series of forecast lead-times is a measure of how well trends in the predicted anomalies follow trends in actual anomalies.
ACC values lie between +1 and -1. Where ACC values:
- approach +1 there is good agreement and the forecast anomaly has had value.
- lie around 0 there is poor agreement and the forecast has had no value.
- approach –1 the agreement is in anti-phase and the forecast has been very misleading.
Where the ACC value falls below 0.6 it is considered that the positioning of synoptic scale features ceases to have value for forecasting purposes.
At ECMWF the anomaly correlation coefficient (ACC) scores represent the spatial correlation between:
- the mean of the anomalies of a forecast product relative to a reference model climate and
- the mean of the anomalies of observations or reanalysis relative to the same reference model climate.
- For medium range forecasts the anomaly correlation coefficient is evaluated between:
- the mean of the anomaly of the forecast product relative to the medium range model climate (M-climate) and
- the mean of the anomaly of the verifying control analysis relative to the medium range model climate (M-climate).
- For sub-seasonal range products the correlation is evaluated between:
- the mean of the anomaly of the forecast product measured relative to the sub-seasonal range model climate (SUBS-M-climate) and
- the mean of the anomaly of the verifying control analysis relative to the sub-seasonal range model climate (SUBS-M-climate).
- For seasonal products the correlation is evaluated between:
- the mean of the anomaly of the forecast product relative to the a model climatology based on the ERA-interim re-analysis (based on the period 1993-2016) and
- the mean of the anomaly of the verifying observations or reanalysis relative to the seasonal model climate (S-M-Climate).
The medium range model climate (M-climate) and sub-seasonal range model climate (SUBS-M-climate) are based on re-forecasts spanning the last 20 years, which used the ERA5 reanalysis for their initialisation
The seasonal model climate (S-M-climate) is based on re-forecasts spanning the last 20 years, which used the ERA-interim re-analysis for their initialisation
As forecast lead-time increases so forecasts progressively correlate less well with observed values. The ensemble mean smooths out many smaller scale features and performs best against the verifying analysis. This supports preferential use of the ensemble mean in practical forecasting. Typically ACC falls to 0.6 at around day 8 or day 9 for ensemble control (CTRL), and at around day 10 for the ensemble mean (EM).
Fig6.2.2-1: Variation of Anomaly Correlation Coefficients for 850hPa temperatures with forecast lead time during the period 1 July 2023 to 30 June 2024. Ensemble Control (red), ensemble mean (green), an individual ensemble member (blue).
The ensemble mean out-performs the ensemble control and is almost 2½ days better than any individual ensemble member (e.g. The ACC of a perturbed ensemble member at Day7 is being attained by ensemble mean at Day9½). These graphs show results from Cy48r1 when the medium range ensemble and ensemble control had a resolution of 18km.
Fig6.2.2-2: Variation of Anomaly Correlation Coefficients for 500hPa geopotential with forecast lead time during the period 1 July 2023 to 30 June 2024. Ensemble Control (red), ensemble mean (green), an individual ensemble member (blue).
The ensemble mean out-performs the ensemble control and is almost 2 days better than any individual ensemble member (e.g. The ACC of a perturbed ensemble member at Day7 is being attained by ensemble mean at Day9). These graphs show results from Cy48r1 when the medium range ensemble and ensemble control had a resolution of 18km.
The ACC scores have steadily improved over the years at all lead-times. This reflects the improvement of input data, analysis techniques and IFS model formulation (See Fig6.2.2-3).
Fig6.2.2-3: Time series of 12 month running mean average ACC scores for HRES 500hPa geoptential forecasts evaluated against operational analyses during the period 1980 to 2024. Forecast lead-times are shown for: 3 days (blue), 5 days (red), 7 days (green) and 10 days (yellow). Scores were averaged over the northern extra-tropics (bold lines) and southern extra-tropics (thin lines). Shading highlights the differences in scores between the two hemispheres. Note non-uniform scale for ACC values.
Fig6.2.2-4: The plot shows the forecast lead-time (in days) at which Anomaly Correlation Coefficient (ACC) of the HRES forecast dropped to 80% for:
- the month mean (blue line, with blue spots at each month),
- twelve-month mean centred on that month (red line).
Verification follows updated WMO/CBS guidelines as specified in the Manual on the GDPFS, Volume 1, Part II, Attachment II.7, Table F, (2010 Edition - Updated in 2012).
Current Anomaly Correlation Coefficient (ACC) scores for northern hemisphere, Southern Hemisphere, tropics and Europe are available.
The score for the northern hemisphere extra-tropics is a primary headline score of the ECMWF Ensemble Control Forecast (ex-HRES).
(FUG Associated with Cy49r1)