...
The headline medium-range ensemble forecast skill score is the maximum lead time (in days), up to 10-days ahead, in which the Continuous Ranked Probability Skill Score (CRPSS) is greater than a value of 0.5, when compared to a simple persistence benchmark forecast using EFAS historical Forced simulation (sfo) as proxy observations. Forecast skill is calculated using river discharge reforecasts for a set of past dates, based on a configuration as close as possible to the operational setting. ECMWF-ENS medium and extended range reforecasts are used and are run twice per week for the past 20-years with 11 ensemble members.
...
A widely used benchmark forecast for short to medium-range forecast skill evaluation is a hydrological persistence forecast (Alfieri et al., 2014; Pappenberger et al., 2015). Here, the 6 hr river discharge value of the EFAS historical Forced simulation (sfo) from the time-step previous to reforecast initilisation is used for all lead-time out to 10-days ahead. For example, for the reforecast initialised on 00UTC 3 January 1999, the mean 6hr river discharge value from 18UTC 2 January 1999 to 00UTC 3 January 1999 is extracted from EFAS historical Forced simulation (sfo) and persisted for all lead times.
...
Forecast skill is evaluated against EFAS historical Forced simulation (sfo), as a proxy to river discharge observations, for n=2651 fixed reporting point stations across the EFAS domain. The advantages advantage of using sfo instead of in situ river discharge observations is that the forecast skill can be determined independently from the hydrological model error and having a complete spatial and temporal coverage, so that forecast skill can be determined across the full EFAS domain. Users must be aware that the key assumption with the proxy observation approach is that the EFAS hydrological model performance, in which sfo is based, is reasonably good for the station of interest. If the hydrological model performance is poor, then particular care must be made in interpreting forecast skill scores.
...
The ensemble forecast performance is evaluated using the Continuous Ranked Probability Score (CRPS) (Hersbach, 2000), one of the most widely used headline scores for probabilistic forecasts. The CRPS compares the continuous cumulative distribution of an ensemble forecast with the distribution of the observations observations. It has an optimum value of 0 and measures the error in the same units as the variable of interest (here river discharge in m3 s-1). It collapses to the mean absolute error for deterministic forecasts (as is the case here for the single-valued persistence benchmark forecast). The CRPS is expressed as a skill score to calculate forecast skill, CRPSS, which measures the improvement over a benchmark forecast and is given in:
...
A CRPSS value of 1 indicates a perfect forecast, CRPSS > 0 shows forecasts more skilful than the benchmark, CRPSS = 0 shows forecasts are only as accurate as the benchmark, and a CRPSS < 0 warns that forecasts are less skilful than the the benchmark forecast. The headline EFAS medium-range forecast skill score uses a CRPSS threshold of 0.5 in the summary layer in the EFAS web map viewer, this can be interpreted as the EFAS forecast has 50% less error than the benchmark forecast.
The CRPSS is calculated with EFAS medium-range reforecasts against a single-valued persistence valued persistence benchmark foreacsts forecasts and verified against EFAS sfo river discharge simulations as proxy observations. CRPSS headline scores are then mapped on the EFAS map viewer, and CRPSS and CRPS time-series plots are produced for each fixed reporting point station.
...