Page History

Table of Contents

Relative Operating Characteristics (ROC)

The Relative Operating Characteristics give a measure of the effectiveness of a forecast system to predict an event that actually happens balanced against forecast of an event that fails to occur.   ROC is derived from measuring the area beneath the results plotted on a ROC diagram.  ROC area values indicate:

• ROC value 1.0 implies consistent Hit Rate and no False Alarms.
• ROC value 0.5 implies no skill with as many Hits as False Alarms. 
• ROC value 0.0 implies consistent False Alarm rate and no Hits.  

Relative Operating Characteristics skill score charts

The Relative Operating Characteristics (ROC) skill score charts give a measure of how well the seasonal (SEAS5) forecast anomalies have represented the observed anomalies.

At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

For clarity of presentation on the charts, the ROC skill score is defined as ROCS = 2(ROC area - 0.5).  So ROC skill scores range between:

• ROC skill score value 1.0 implies forecasts with high skill.
• ROC skill score value 0.0 implies forecasts with no skill forecasts. 
• ROC skill score value -1.0 implies misleading forecasts.  

  The colouring on the charts gives the measure of success of the SEAS5 forecast in representing the actual conditions: 

• Brown, red and dark orange colouring imply seasonal (SEAS5) forecast anomalies have shown skill in representing the tercile of the observed anomalies.
• Orange colouring implies seasonal (SEAS5) forecast anomalies no skill in representing the tercile of the observed anomalies
• Blue colouring implies seasonal (SEAS5) forecast anomalies have had no success or been misleading in representing the tercile of the observed anomalies

Fig8.3.3-1: Chart of ROC skill score for 500hPa geopotential height for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

• Positive values (yellow, orange, red, brown) indicate success of the SEAS5 forecast in representing the observed tercile.
• Negative values (grey and blues) indicate no success (or  misleading) SEAS5 forecast in representing the observed anomalies.

Black dots signify 95% confidence.

Fig8.3.3-2: Chart of ROC skill score for 2m temperature for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

• Positive values (yellow, orange, red, brown) indicate success of the SEAS5 forecast in representing the observed tercile.
• Negative values (grey and blues) indicate no success (or  misleading) SEAS5 forecast in representing the observed anomalies.

Black dots signify 95% confidence.

Fig8.3.3-3: Chart of ROC skill score for sea surface temperature for the verifying period Sep, Oct, Nov derived from SEAS5 runs with data time of early Aug.  At each location, the predicted probability of an event lying in the indicated tercile (upper, lower) is compared with frequency that the forecast event verifies within the indicated tercile (upper, lower).         

• Positive values (yellow, orange, red, brown) indicate success of the SEAS5 forecast in representing the observed tercile.
• Negative values (grey and blues) indicate no success (or  misleading) SEAS5 forecast in representing the observed anomalies.

Black dots signify 95% confidence.