The distinction between validation (forecast system’’s characteristics) and verification (forecast system’’s predictive skill) is as relevant in probabilistic as in deterministic forecasting. Some statistical concepts to facilitate the use and interpretation of probabilistic medium-range forecasts - ensemble forecasts - are given below. As with the deterministic forecasting system, probability verification can address the accuracy (how close the forecast probabilities are to the observed frequencies), the skill (how the probability forecasts compare with some reference system) and utility (the economic or other advantages of the probability forecasts).
Sceptics of probability forecasts argue that forecasters might exaggerate uncertainty ““to cover their backs””. However, as will be shown, the verification of probabilistic forecasts takes the ““reliability”” of the probabilities into account and will detect any such misbehaviour. Indeed, one of the most used verification scores, the Brier score, is mathematically constructed in a way that encourages forecasters to state the probability they really believe in, rather than some misperceived ““tactical”” probability.
This chapter discusses only the most commonly used probabilistic validation and verification methods. For a full presentation the reader is referred to Nurmi (2003).
The most transparent way to illustrate the performance and characteristics of a probabilistic forecast system is the reliability diagram, where the x-axis is the predicted probability and the y-axis the frequency with which the forecasts verify. It serves both as a way to validate the system and to verify its forecasts (see Fig12.B.1).
Fig12.B.1: Schematic explanation of the reliability diagram. Out of thirty 20% probability forecasts, the predicted event should verify six times (i.e. in 20% of the time) not more, not less.
When the forecast probabilities agree with the frequency of events for this particular probability the distribution should lie along the 45° diagonal. In such a case the probability forecasts are considered reliable. The frequency of forecast probabilities is represented by green circles of varying sizes (see Fig12.B.2).
Fig12.B.2: Probabilities with good sharpness. The probabilities cluster as far away as possible from the climatological frequency, here assumed to be 50%.
Climatological probability averages used as forecasts would yield perfect reliability, since the distribution would be exactly on the 45° diagonal, but would not be very useful. Ideally, we want the forecast system, while mainly reliable, to span as wide a probability interval as possible, with as many forecasts as possible away from the climatological average and as close to 0% and 100% as possible. The property of a probabilistic forecast system to spread away from the climatological average is called sharpness. The distribution in Fig12.B.2 has good sharpness, whereas it is poor in Fig12.B.3.
Fig12.B.3: An example of a reliable forecast system with poor sharpness. When the predictability of a certain weather parameter is low, the forecasts might still be reliable but will tend to cluster around the climatological average.
Improvements in probability forecasts, provided they are reliable, will be accompanied by improved sharpness until, ultimately, only 0% and 100% forecasts are issued and verify, corresponding to a perfect deterministic forecast system. However, an improvement in sharpness does not necessarily mean that the forecast system has improved.
Most probabilistic forecast systems, both subjective and objective, tend to give distributions flatter than 45°. This means that low risks are underestimated and high risks overestimated - the forecast system is overconfident. Fig12.B.4 shows an example of overconfidence.
Fig12.B.4: Probability forecasts with good sharpness but over-confident. Forecasts of 0% probability of rain are not always verified as dry; forecasts of 100% probability of rain are not always derified as wet.
The less common case: the distribution is steeper than 45°, low risks have been overestimated and high risks underestimated. The forecast systems are then under-confident (see Fig12.B.5).
Fig12.B.5: Probabilities indicating reluctance to use very high or low probabilities. The probability forecasts are under-confident.
Under- or overconfidence can be corrected by the calibration of probabilities (see B-5).
A more detailed way of validating the spread is by a rank histogram (sometimes called a Talagrand diagram). It is constructed from the notion that in an ideal ensemble system the verifying analysis is equally likely to lie in any ““bin”” defined by any two ordered adjacent members, including when the analysis is outside the ensemble range on either side of the distribution. This can be understood from induction, if we consider an ideal ensemble with one, two or three members:
With one ensemble member ( I ) verifying observations (
) will always (100%) fall ““outside”” I
With two ensemble members (I I), verifying observations will for this ideal ensemble fall outside in two cases out of three II
With three ensemble members (I I I), verifying observations will fall for this ideal ensemble outside in two cases out of four III
In general, if N = number of members, the verification will in two cases out of N + 1 always fall outside, yielding a proportion of 2 / (N + 1) outside. For the same reasons the HRES and the ENS Control should lie outside the ensemble 2 / (N + 1) of the time. For a 50-member ensemble system this means 4%. This is consistent with the consideration of probability that, due to the limited number of ensemble members, it would be unrealistic to assume that the probability was 0% or 100% just because none or all of the members forecast the event.
In an ideal ensemble, the rank histogram distribution should, on average, be flat with equal numbers of verifying observations in each interval. If there is a lack of spread, this will result in a U-shaped distribution with an over-representation of cases where the verifications fall outside the ensemble and under-representation of cases when they fall within the ensemble centre. If the system has a bias with respect to the verifying parameter, the U-shape might degenerate into a J-shape.
An ideal ensemble system might, however, display a U-shape distribution due to observation uncertainties. For example, with 50 ensemble members an ensemble spread of 20°C yields an average bin width of 0.4°C, an ensemble spread of 5°C yields an average bin width of only 0.1°C, smaller than the observation uncertainty (see Fig12.B.6). The small bin size introduces an element of chance with respect to which bin the observation will fall into. The bin sizes, due to the normal distribution, will increase with increasing distance from the centre, and an observation is more likely to end up in a bin further away from the centre than closer to the centre. This will result in a misleading U-shaped distribution.
Fig12.B.6: An observation (a filled circle) and its uncertainty assumed symmetric (the arrows). The forecast bins widen their intervals away from the centre (the mean of the distribution, here off the diagram to the right), so an observation is more likely, for random reasons, to fall into an outer and wider bin than an inner and narrower one.
In contrast to deterministic forecasts, an individual probabilistic forecast can never be ““right”” or ““wrong”” except when 0% or 100% have been stated. Probability forecasts can therefore, only be verified from large samples of forecasts.
The most common verification method for probabilistic forecasts, the Brier score (BS) or the Mean Square Error of Probability Forecasts, has a mathematical structure similar to the Mean Square Error (MSE).
BS measures the difference between the forecast probability of an event (p), and its occurrence (o) expressed as 0 (event did not occur) or 1 (event did occur). As with RMSE, the BS is negatively orientated (i.e. the lower, the better”).
The Brier Score (BS) measures the mean squared probability error of the forecast against a reference dataset (observations, or analyses, or climatology)
The Brier Score (BS) is a measure of how good forecasts are in matching observed outcomes. Where:
The Brier Skill Scores (BSS) measures the relative skill of the forecast compared to climatology.
BSS is evaluated for each forecast centre by calculating the function BSS = 1 - BSforecast / BSclimat
Where:
Similar to the MSE, the BS can be decomposed into three terms, the most often quoted was suggested by Allan Murphy (1973, 1986) who used ““binned”” probabilities:
where nk is the number of forecasts of the same probability category k.
The first term measures the reliability (i.e. how much the forecast probabilities can be taken at face value). On the reliability diagram this is the nk weighted sum of the distance (vertical or horizontal) between each point and the 45° diagonal (see Fig12.B.7).
The second term measures the resolution (i.e. how much the predicted probabilities differ from a climatological average and therefore contribute information). On the reliability diagram this is the weighted sum of the distances to a horizontal line defined by the climatological probability reference (see Fig12.B.7).
The third term measures the uncertainty (i.e. the variance of the observations). It takes its highest, most ““uncertain””, value when õ = 0.5 (see Fig12.B.8).
Fig12.B.7: Summary of Allan Murphy’’s reliability and resolution terms.
Fig12.B.8: Uncertainty is at its maximum for a climatological observed probability average of 50%.
Resolution, the degree to which the forecasts can discriminate between more or less probable events, should not be confused with sharpness, the tendency to have predictions close to 0% and 100%. Resolution and Sharpness are also independent of one another.
The same ““uncertainty”” can be understood from the familiar fact that it is easier to predict the outcome of tossing a coin if it is heavily biased. In the same way, if it rains frequently in a region and rarely stays dry, forecasting rain can be said to be ““easier”” than if rain and dry events occur equally often. The uncertainty is purely dependent on the observations, just as the Aa term in the RMSE decomposition. It is also the BS of the sample climatology forecast and plays the same role with the BS as the Aa term with the RMSE (see Forecast Error Baseline). Comparisons of Brier scores for different forecast samples can only be made if the uncertainty is the same.
The Brier score is strictly ““proper”” (i.e. it encourages forecasters to really try to find out the probability, without thinking about whether the forecast value is ““tactical”” or not). Indeed, if forecasters deviate from their true beliefs, the BS will ““punish”” them! This sounds strange. How can an abstract mathematical equation know someone’’s inner beliefs?
Assume forecasters honestly think the probability of an event is p but have, for misguided ““tactical”” reasons, instead stated r. If the event occurs, the contribution to the BS (first term) is (1 - r)2 weighted by the probability for the outcome to occur. If the event does not occur the contribution to the BS (first term) is (r - 0)2 weighted by the probability for the outcome not to occur. For these weightings, the ““honest”” probability must be used; p when the event occurs, (1 - p) when the event does not occur. This is where the forecaster’’s true beliefs are revealed!
The expected contribution to the BS is therefore:
Differentiating with respect to r yields
with a minimum for r = p. Therefore to minimize the expected contribution to the Brier Score the honestly believed probability value should be used.
A Brier Skill Score (BSS) is conventionally defined as the relative probability score compared with the probability score of a reference forecast.
““Uncertainty”” plays no role in the BSS.
Probabilities often refer to the risk that some threshold might be exceeded, for example that the precipitation >1 mm/12hr or that the wind >15 m/s. However, when evaluating a probabilistic system, there are no reasons why these thresholds are particularly significant. For the Rank Probability Score (RPS) the BS is calculated for different (one-sided) discreet thresholds and then averaged over all thresholds.
Discrete Ranked Probability Score (RPS) is the deviation of the forecast values being placed within a category against corresponding observations that actually lie within that category (e.g. tercile, quintile, etc.). The words "discrete" and "ranked" refer to the discrete nature of the categories.
The Ranked Probability Score (RPSS) is a measure of how good forecasts are in matching observed outcomes. Where:
Ranked Probability Skill Score (RPSS) is evaluated for by calculating the function RPSS = 1 − RPSforecast / RPSreference where:
Where:
Weigel, A.P., D. Baggenstos, M. A. Liniger, F. Vitart, and F. Appenzeller, 2008: Probabilistic verification of monthly temperature forecasts. Mon. Wea. Rev., 136, 5162‐5182.
A generalisation of Ranked Probability Score (RPS) is the Continuous Rank Probability Score (CRPSS) where the thresholds are continuous rather than discrete (see Nurmi, 2003; Jollife and Stephenson, 2003; Wilks, 2006). The Continuous Ranked Probability Score (CRPS) is a measure of how good forecasts are in matching observed outcomes. Where:
CRPS is calculated by comparing the Cumulative Distribution Functions (CDF) for the forecast against a reference dataset (observations, or analyses, or climatology) over a given perio
The Continuous Ranked Probability Skill Score (CRPSS) is a measure of how good forecasts are in matching observed outcomes. Where:
CRPSS is evaluated by calculating the function CRPSS = 1 − CRPSforecast / CRPSclimat where:
Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559-570.
A powerful way to verify probability forecasts and in particular to compare their performance with deterministic forecast systems, is the two-dimensional ““Relative Operating Characteristics”” or ““ROC”” diagram. These categorical forecasts will produce a set of pairs of ““Hit Rate”” and ““False Alarm Rate”” values to be entered into the ROC diagram: False Alarm Rate (FR) on the x-axis and Hit Rate (HR) value on the y-axis (derived from the Contingency Table). The upper left corner of the ROC diagram represents a perfect forecast system (no false alarms, only hits). The closer any verification is to this upper left corner, the higher the skill. The lower left corner (no false alarms, no hits) represents a system which never warns of an event. The upper right corner represents a system where the event is always warned for (see Fig12.B.9).
Fig12.B.9: The principle of the ROC diagram: a large number of probability forecasts are turned into categorical forecasts depending on whether the probability values of individual forecasts are above or below a certain threshold. The false alarm rate and the hit rate are calculated, thus determining the position in the diagram (red filled circle).
Probabilistic forecasts are transformed into categorical yes/no forecasts defined by thresholds varying from 0% to 100% (see Fig12.B.10).
Fig12.B.10: The same as Fig12.B.9, but repeated for several thresholds between 0 and 100%, including the hit rates and false alarm rates of the deterministic model; although not providing probabilistic predictions it can be represented on the diagram by its typical hit rate and false alarm rate (green filled circle).
The ROC score is the area underneath the forecast curve (see Fig12.B.11).
Fig12.B.11: The area underneath the points, joined by straight lines, defines the ROC area, which is, ideally, 1.0 and at worst 0.0. Random forecasts yield 0.5, the triangular area underneath the 45° line.
There are two schools on how to calculate this: either with a smooth spline or linearly, connecting the points.
For operational purposes the reliability can be improved by calibration using verification statistics (e.g. if it is found that in cases when 0% has been forecast, the event tends to occur in 30% of the cases, and when 100% has been forecast, the event tends to occur only in 70% of the cases). If the misfit is linearly distributed in between these two extremes, the reliability can be made perfect by calibration - but at the expense of reduced sharpness, since very low and very high probabilities are never forecast (see Fig12.B.12).
Fig12.B.12: The probability distribution from an overconfident forecast system is calibrated, limiting the range from 0% - 100% to 30% - 70%.
On the other hand, when the probability forecasts are under-confident, calibration might restore the reliability without giving up the sharpness (see Fig12.B.13).
Fig12.B.13: The probabilities from an under-confident forecast system are calibrated, widening the range from 30% to 70% to 0% to 100%.
This means that there is some ““hidden skill”” in probability forecasts biased in this way.
An efficient way to improve the ensemble forecast, both the EM and the probabilities, is by Statistical Post-Processing (SPP), which is an advanced form of calibration of the output from the deterministic ensemble members. The most commonly used SPP method is ““model output statistics”” (MOS).
Deterministic NWP forecasts are statistically matched against a long record of verifying observations though a linear regression scheme. The predictand (Y) is normally scalar (for example 2m temperature) and the predictors (X) one or several forecast parameters, selected by a linear regression system using the parameters which provide the most information (e.g. forecasts of 2m temperature use 850hPa temperature, 500hPa geopotential etc):
Y = X1 + X2 ·T2m +X3 ·T850hPa +X4 ·Z500 + ……etc
where the coefficients Xi=1, 2, 3 ... n are estimated by the regression scheme. For this discussion it is sufficient to consider the simple MOS equation:
Y = X1 + X2·T2m
where X1 and X2 have been estimated from a large amount of representative historical material. This is often quite an effective correction equation, since the errors in many meteorological forecast parameters in a first approximation tend to be linearly dependent on the forecast itself (except perhaps precipitation and cloudiness).
The MOS equation not only minimizes the RMSE, it also corrects simultaneously for both systematic mean errors and for the variability. In the above equation X1 represents the mean error correction and X2 the variability correction. There is therefore no necessity to apply two different schemes, one for reducing the systematic error (““bias””) and one for correcting the spread.
However, the corrections imposed by MOS have different emphases in the short and medium range. In the short range, where most synoptic features are forecast with realistic variability, the MOS equation mainly corrects true systematic errors and representativeness errors.
Fig12.B.14: A scatter diagram of forecast errors versus forecast for Tromsöö in northern Norway, November 2010 - February 2012. Cold temperatures are too cold and, as a whole, the forecasts overestimate the variability of the temperature.
The scatter diagram in Fig12.B.14 depicts the errors at D+1 and therefore shows true systematic errors: the colder the forecast, the larger the mean error, which is equivalent to over-forecast variability.
MOS also improves forecasts in the medium range but, with increasing forecast range, less and less of this improvement is due to the MOS equation’’s ability to remove systematic errors. In the medium range, the dominant errors are non-systematic. These non-systematic errors (e.g. false model climate drift) can appear as false systematic errors (e.g. see Fig12.A.4). They will thus be ““corrected”” by the MOS in the same way as true systematic errors. By this means MOS is essentially dampening the forecast anomalies and thereby minimizing the RMSE. This might be justified in a purely deterministic context but not in an ensemble context, where the most skilful damping of less predictable anomalies is achieved by ensemble averaging through the EM. It is therefore recommended that MOS equations are calculated in the short range, typically at D+1, based on forecasts from the CTRL, and then applied to all the members in the ensemble throughout the whole forecast range, as long as any genuine model drift can be discarded.
About every few years, NWP models undergo significant changes that make the MOS regression analysis obsolete. There are, however, techniques whereby the MOS can be updated on a regular (monthly or quarterly) basis, although this does not completely eliminate the drawback of historic inertia. Alternatively, adaptive methods have increasingly come into use. Here the coefficients X1 and X2 in the error equation are constantly updated in the light of daily verification (Persson, 1991).
Fig12.B.15 shows forecasts and observations for the location with severe systematic 2m temperature errors depicted in Fig12.B.14. It is not a case of ““plain bias”” but of ““conditional bias””, since mild forecasts are less at error than cold forecasts. A simple mean-error correction would therefore not be optimal.
Fig12.B.15: Adaptive Kalman filtering of 2-metre temperature forecasts for Tromsöö in northern Norway during winter 2012. The forecasts are too cold and over-variable, both of which are remedied by X1 and X2 in a 2-parameter error equation.
By a daily verification, the Kalman filter estimates the coefficients X1 and X2 in theerror equation:
Err = X1 + X2·Tfc
where Tfc is the verified forecast. The coefficients are updated from a variational principle of ““least effort””, whereby the equation line is translated (by modifying X1) and rotated (by modifying X2), so that it takes the verification into account, considering the uncertainties in the verification and the coefficients (see Fig12.B.16).
Fig12.B.16: A schematic illustration of the workings of an adaptable MOS by Kalman filtering. At a given time the error equation has a certain orientation (full red line) with a certain estimated uncertainty (red dashed lines). A forecast is verified and yields an error (red filled circle) that does not normally fall on the error line. Depending on the interplay between the equation uncertainty and the verification uncertainty (dashed red circle), the error equation line is translated and rotated to take the new information into account, after which this information is discarded.
Note that the system keeps information only about the error equation and its uncertainty and the last, not yet verified forecast. When the forecast is verified and the verification has affected the error equation, the verifying observation is discarded.
Fig12.B.17 shows an ensemble forecast for the same location with severe systematic 2m temperature errors.
Fig12.B.17: A plume diagram for Tromsöö, 12 February 2012. The forecast is too cold with 50-100% probabilities of temperatures < -15°C.
The two-dimensional error equation is able to apply corrections which are different for different forecast temperatures and thus take the flow dependence into account to some degree. The error equation is applied to all ensemble members at all ranges, assuming no significant model drift (see Fig12.B.18).
Fig12.B.18: shows an ensemble forecast for the same location with severe systematic 2m temperature errors.: The same as Fig12.B.17 but after the Kalman-filtered errors equation has been applied. Mild forecasts have hardly been modified, whereas cold ones have been substantially warmed, leading to less spread and more realistic probabilities (e.g. 0% probabilities for 2m temperature <- 15°C).
A two- or multi-dimensional error equation is able not only to correct for mean errors, but also systematic over- and under-forecasting of the variability, thereby providing realistic probabilities.