Page History

...

Using Verification Metrics alongside the Forecast

It is useful to have some measures of the current performance and biases of the IFS.  Users can assess from Reliability and ROC diagrams whether the forecast model is:

...

A common feature of reliability diagrams is that the profile of the forecasts (red line in Fig1) has a shallower slope than the diagonal, but crosses it somewhere near the climatological value (blue line intersection).  This means that the forecast has a tendency to be over-confident.  Users should adjust forecast probabilities, even if departures from the diagonal are only small, to offset the tendency to over-forecast frequently observed events and to under-forecast rather more rare events.

Current Reliability Diagrams (which include distribution of forecast probabilities) are available on Opencharts (days 4, 6, and 10 only)

The ROC diagram

...

A system which always forecast climatological probabilities, for example, would have no discrimination ability (i.e. zero resolution).  The resolution can be investigated using the Relative Operating Characteristic (ROC) diagram, which plots hit rate on the y-axis against false alarm rate on the x-axis.   Ideally the Hit Rate should be high and the False Alarm Rate low (i.e. ideally the graph should lie well towards the top left corner) and generally the Hit Rate should be better than the False Alarm Rate (i.e. values should lie above the diagonal).  

...

The ROC score is the area beneath the graph on the ROC diagram and lies between 1 (perfect capture of events) and 0 (consistently warning of events that don't happen).  Fig1 shows high effectiveness in forecasting events (ROC score 0.859) while Fig2 shows reduced effectiveness (ROC score 0.593).  This is as expected as the forecast range becomes longer. 

Current ROC Diagrams are available on Opencharts (for day5 onwards).

Fig1: Reliability Diagram (left) and ROC diagram (right) regarding lower tercile for T2m in Europe area for week1 (day5-11), DT:20 Jun 2019.

...

• • the  anomaly of the product measured relative to the a model climatology based on the ERA-interim re-analysis (based on the period 1993-2016) and
  • the anomaly of the verifying observations or reanalysis relative to the seasonal model climate (S-M-Climate). 

The seasonal model climate (S-M-climate) is based on re-forecasts spanning the last 20 years, which used the ERA-interim reanalysisre-analysis for their initialisation

Anomaly correlation coefficient (ACC) charts are produced for several parameters.  Each chart shows the skill of the forecast at each location for the given month and lead-time.  

...