Using verification metrics with the output

Using Verification Metrics alongside the Forecast

When a forecaster uses long-range (seasonal) forecasts it is vital that they understand, from the outset, the general characteristics of the model forecasts relative to the subsequent verifying observations; for example whether or not the model typically over-forecasts or under-forecasts certain types of outcome.  They should then interpret any forecast signals accordingly. Usually this will mean that they need to be wary of over-stating the significance of any such signals (that have historically been unreliable and/or unskilful).  Such a strategy should be applied at all lead times, within forecasting in general. However it is particularly important for the longer lead forecasts (such as monthly and seasonal).

ECMWF provides a number of verification metrics to use in this way, such as anomaly correlation coefficients, reliability diagrams and ROC curves, which have all been computed using the re-forecasts.

Reliability Diagrams - capacity to descriminate between model over- or under-forecasting

In a reliability diagram, the plotted points will lie to the right of the diagonal if there is over-forecasting (e.g. where rain is forecast with 100% probability and rain has actually been observed on only 80% of occasions) and to the left where under-forecasting.  Ideally points should lie on the diagonal.  The size of the departure from the diagonal indicates the magnitude of the over- or under-forecasting error.  A common feature of reliability diagrams is that the profile of the forecasts (line joining the red dots on the examples below) has a shallower slope than the diagonal, but crosses it somewhere near the climatological value (blue line intersection). This means that the forecast has a tendency to be over-confident, and that the signals it shows need to be "watered down" (even if they are only small).

Fig8.3.1: Reliability diagrams for 2m temperature based on July starts of the long-range forecasts for months 4-6.

• left for the tropics - a slight tendency towards over-confidence, more especially where forecasting that this event (warm anomalies) will happen.
• right for Europe - a tendency towards over-confidence, though the sample size for high confidence forecasts is small, making the plot noisy.

Fig8.3.2: Reliability diagrams for rain based on July starts of the long-range forecasts for months 4-6:

• left for the tropics - a tendency towards over-confidence.
• right for Europe - forecast not reliable at all (so should not be used, unless there are exceptional circumstances that warrant an expectation of skill that is ordinarily not there).

ROC diagrams - capacity to discriminate when events are more likely to happen

It is also necessary to assess the extent to which the long-range (seasonal) forecasting system has the capacity to successfully discriminate occasions when an event is likely from occasions when it is not.  This is also known as the 'resolution' of the forecasting system (not to be confused with spatial and temporal resolution!).  A system which always forecast climatological probabilities, for example, would have no discrimination ability (i.e. zero resolution).  The resolution can be investigated using the Relative Operating Characteristic (ROC) diagram, which plots hit rate on the y-axis against false alarm rate on the x-axis.   Ideally the Hit Rate should be high and the False Alarm Rate low (i.e. ideally the graph should lie well towards the top left corner) and generally the Hit Rate should be better than the False Alarm Rate (i.e. values should lie above the diagonal).
 

Fig8.3.3: ROC diagrams for Europe based on July starts of the long-range forecasts for months 4-6:

• left for temperatures in the upper tercile - the Hit Rate is slightly better than the False Alarm Rate indicating that the forecast system has some limited ability to discriminate occasions when warm events are likely from occasions when they are not. 
• right for precipitation in the upper tercile - the Hit Rate and False Alarm Rate are similar throughout indicating that the seasonal forecast system has no ability to distinguish occasions when it will be wet from occasions when it will not.
  
  

Anomaly Correlation - capacity to assess skill of the forecast

The Anomaly Correlation Coefficient (ACC) is the spatial correlation between the forecast anomaly and the verifying analysis anomaly, where the anomalies are computed with respect to ERA-Interim.   ACC charts are produced for several parameters to enable users to identify areas where the seasonal model forecast values at each location for the given month and lead-time exhibit skill.  High ACC implies a strong correlation between forecast anomalies and the verifying observed anomalies; zero ACC implies the forecasts are no better than climatology; areas of negative ACC (shaded blue) imply the forecasts have a tendency to predict the opposite of what subsequently happened (though often this is a sampling issue which should just be interpreted as "no skill").  Locations with correlation significantly (95% confidence level) different from zero are highlighted by dots.

Fig8.3.4: Anomaly Correlation Coefficient for 2m temperature for months 2-4 based on November runs.  High ACC (red) over the eastern Pacific suggests the seasonal model captures the amplitude of the variability of the 2m temperature quite well, Grey over Siberia suggests the model is no better than climatology (i.e. doesn't capture the variability); and cyan near Newfoundland suggests the seasonal model can be rather unreliable and misleading in this area.