When a forecaster uses long-range (seasonal) forecasts it is vital that they understand, from the outset, the general characteristics of the model forecasts relative to the subsequent verifying observations; for example whether or not the model typically over-forecasts or under-forecasts certain types of outcome. They should then interpret any forecast signals accordingly. Usually this will mean that they need to be wary of over-stating the significance of any such signals (that have historically been unreliable and/or unskilful). Such a strategy should be applied at all lead times, within forecasting in general. However it is particularly important for the longer lead forecasts (such as monthly and seasonal).
ECMWF provides a number of verification metrics to use in this way, such as anomaly correlation coefficients, reliability diagrams and ROC curves, which have all been computed using the re-forecasts.
In a reliability diagram, the plotted points will lie to the right of the diagonal if there is over-forecasting (e.g. where rain is forecast with 100% probability and rain has actually been observed on only 80% of occasions) and to the left where under-forecasting. Ideally points should lie on the diagonal. The size of the departure from the diagonal indicates the magnitude of the over- or under-forecasting error. A common feature of reliability diagrams is that the profile of the forecasts (line joining the red dots on the examples below) has a shallower slope than the diagonal, but crosses it somewhere near the climatological value (blue line intersection). This means that the forecast has a tendency to be over-confident, and that the signals it shows need to be "watered down" (even if they are only small).
Fig8.3.1: Reliability diagrams for 2m temperature based on July starts of the long-range forecasts for months 4-6.
Fig8.3.2: Reliability diagrams for rain based on July starts of the long-range forecasts for months 4-6:
It is also necessary to assess the extent to which the long-range (seasonal) forecasting system has the capacity to successfully discriminate occasions when an event is likely from occasions when it is not. This is also known as the 'resolution' of the forecasting system (not to be confused with spatial and temporal resolution!). A system which always forecast climatological probabilities, for example, would have no discrimination ability (i.e. zero resolution). The resolution can be investigated using the Relative Operating Characteristic (ROC) diagram, which plots hit rate on the y-axis against false alarm rate on the x-axis. Ideally the Hit Rate should be high and the False Alarm Rate low (i.e. ideally the graph should lie well towards the top left corner) and generally the Hit Rate should be better than the False Alarm Rate (i.e. values should lie above the diagonal).
Fig8.3.3: ROC diagrams for Europe based on July starts of the long-range forecasts for months 4-6:
Anomaly Correlation Coefficient (ACC) charts show the correlation, for all geographical locations, in map form, between the forecast anomaly (in the re-forecast dataset) and the verifying analysis anomaly, where anomalies are computed with respect to ERA-Interim. ACC charts are produced for several parameters to enable users to identify areas where the seasonal model forecast values at each location for the given month and lead-time exhibit skill. High ACC implies a strong correlation between forecast anomalies and the verifying observed anomalies; zero ACC implies the forecasts are no better than climatology; areas of negative ACC (shaded blue) imply the forecasts have a tendency to predict the opposite of what subsequently happened (though often this is a sampling issue which should just be interpreted as "no skill"). Locations with correlation significantly (95% confidence level) different from zero are highlighted by dots.
Fig8.3.4: Anomaly Correlation Coefficient for 2m temperature for months 2-4 based on November runs. High ACC (red) over the eastern Pacific suggests the seasonal model captures the amplitude of the variability of the 2m temperature quite well, Grey over Siberia suggests the model is no better than climatology (i.e. doesn't capture the variability); and cyan near Newfoundland suggests the seasonal model can be rather unreliable and misleading in this area.