Page History

...

The same ““uncertainty”” can be understood from the familiar fact that it is easier to predict the outcome of tossing a coin if it is heavily biased.  In the same way, if it rains frequently in a region and rarely stays dry, forecasting rain can be said to be ““easier”” than if rain and dry events occur equally often.  The uncertainty is purely dependent on the observations, just as the A_a term in the RMSE decomposition.  It is also the Brier Score of the sample climatology forecast and plays the same role with the Brier Score as the A_a term with the RMSE (see Forecast Error Baseline).  Comparisons of Brier Scores for different forecast samples can only be made if the uncertainty is the same.

...

In general, if N = number of members, the verification will in two cases out of N + 1 always fall outside, yielding a proportion of 2 / (N + 1) outside.  For the same reasons the HRES and the ENS Control should lie outside the ensemble 2 / (N + 1) of the time.  For a 50-member ensemble system this means 4%.  This is consistent with the consideration of probability that, due to the limited number of ensemble members, it would be unrealistic to assume that the probability was 0% or 100% just because none or all of the members forecast the event.

...

A powerful way to verify probability forecasts and in particular to compare their performance with deterministic forecast systems, is the two-dimensional ““Relative Operating Characteristics”” or ““ROC”” diagram.  These categorical forecasts will produce a set of pairs of ““Hit Rate”” and ““False Alarm Rate”” values to be entered into the ROC diagram: False Alarm Rate (FR) on the x-axis and Hit Rate (HR) value on the y-axis (derived from the Contingency Table).  The upper left corner of the ROC diagram represents a perfect forecast system (no false alarms, only hits).  The closer any verification is to this upper left corner, the higher the skill.  The lower left corner (no false alarms, no hits) represents a system which never warns of an event.  The upper right corner represents a system where the event is always warned for (see Fig12.B.12).

...