Page History

...

The diagram shows the relationship between forecast probability of an event and the observed frequency of that event as measured by c(e.g. the probability, measured as the proportion of ENS ensemble members, that a 2m temperature will be greater than 20C, plotted against the climatological (taken from re-forecasts) frequency of that event).  The plotted points lie to the right of the diagonal if there is over-forecasting (e.g. where rain is forecast with 100% probability and rain has actually been observed on only 80% of occasions) and to the left where under-forecasting.  Ideally points should lie on the diagonal.  The size of the departure from the diagonal indicates the magnitude of the over- or under-forecasting error.

...

Fig3: Example of Reliability Diagrams from Opencharts.  Total 24hr precipitation Day6, assessed from ENS ensemble probability forecasts during a three month period and compared climatology from the same period. The traces show the comparison of forecast probabilities against observed occurrences for 24h precipitation totals of >1mm, >5mm, >10mm, >20mm.  Ideally the traces should lie along the dashed blue line (i.e. the ENS ensemble probability forecast probability should agree with the observed frequency).  The diagram shows:

• reasonably good forecasting at low ensemble probabilities
  • e.g. ENS ensemble 20% probability occurred on 20% of the time for each group
• over-forecasting at higher ENS ensemble probabilities:
  • e.g. ENS ensemble 90% probability of >1mm/24h actually occurred only 60% of the time - the wide distribution of forecast probabilities suggest some confidence in the Reliability trace. 
  • e.g. ENS  ensemble 90% probability of >20mm/24h actually occurred 80% of the time - but the very few forecasts of high probabilities suggest very low confidence in the corresponding implied reliabilities.

...

Fig4: Example of Reliability Diagrams from Opencharts.  Temperature anomaly Day4, assessed from ENS ensemble probability forecasts during a three month period and compared climatology from the same period. The traces show the comparison of forecast probabilities of anomalies against observed occurrences of anomalies for 2metre temperature of >8°C below, >4°C below, >4°C above, >8°C above climatology.  Ideally the traces should lie along the dashed blue line (i.e. the ENS ensemble forecast probability should agree with the observed frequency).  The diagram shows:

• under-forecasting at low probabilities
  • e.g. for >8°C above climatology, >4°C above, >8°C below climatology, ENS ensemble 20% probability actually occurred on 35% of the time.
  • e.g. for >4°C below climatology ENS ensemble 20% probability actually occurred on 25% of the time - fairly good correspondence.
• over-forecasting at higher ENS ensemble probabilities e.g.:
  • for >4°C below climatology ENS ensemble 90% probability actually occurred only 70% of the time - the wide distribution of forecast probabilities suggest some moderate confidence in the implied reliability. 
  • for >8°C above climatology ENS ensemble 90% probability actually occurred only 65% of the time - but the very few forecasts of high probabilities suggest very low confidence in the implied reliability.

...

• • for >4°C above climatology ENS ensemble 90% probability actually occurred 85% of the time - the wide distribution of forecast probabilities suggest some moderate confidence in the implied reliability. 
  • for >8°C below climatology ENS ensemble 90% probability actually occurred 85% of the time - but the very few forecasts of high probabilities suggest very low confidence in the implied reliability

...