Page History

...

• ROC value 1.0 implies consistent Hit Rate and no False Alarms.
• ROC value 0.5 implies no skill with as many Hits as False Alarms. 
• ROC value 0.0 implies consistent False Alarm rate and no Hits.

An explanation of ROC diagrams is given in the annex to this guide.

Image Added

Fig8.3.3-1: ROC diagram for 850hPa temperature for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  Points above the lower left/upper right diagonal signify skill.  The ROC score or value is the area beneath the red curve.

Reliability diagrams

The Reliability diagrams give a measure of the tendency of the forecast system to over- or under-forecast and event.  The diagram plots the frequency of a forecast probability of an event against the frequency that the event occurs.   Ideally these should match and is shown by the diagonal line. Where the plot lies:

• below the diagonal implies over-forecasting.  The forecast probability is greater than the observed frequency.
• above the diagonal implies under-forecasting.  The forecast probability is less than the observed frequency.

An explanation of reliability diagrams is given in the annex to this guide.

Image Added

Fig8.3.3-2: reliability diagram for precipitation for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  Points above the diagonal signify under-forecasting, points below the diagonal signify over-forecasting.

Relative Operating Characteristics skill score charts

...

• Brown, red and dark orange colouring imply seasonal (SEAS5) forecast anomalies have shown skill in representing the tercile of the observed anomalies.
• Orange colouring implies seasonal (SEAS5) forecast anomalies no skill in representing the tercile of the observed anomalies
• Blue colouring implies seasonal (SEAS5) forecast anomalies have had no success or been misleading in representing the tercile of the observed anomalies

Fig8.3.3-13: Chart of ROC skill score for 500hPa geopotential height for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

...

Black dots signify 95% confidence.

Fig8.3.3-24: Chart of ROC skill score for 2m temperature for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

...

Black dots signify 95% confidence.

Fig8.3.3-35: Chart of ROC skill score for sea surface temperature for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

...