EFAS v5.1 - Hydrological Post-Processing Calibration Evaluation

As part of the EFAS v5.1 upgrade the hydrological post-processing calibration procedure was performed in January 2024 with the EFAS v5 hydrological reanalysis (water balance) and observations up to 21 January 2024. A total of 1662 stations were calibrated at 6-hourly timesteps and a further 172 were calibrated at daily timesteps. As part of the calibration process an evaluation is performed for each station to assess the expected skill of the post-processing method. This page summarises the results of the evaluation of this calibration for all stations.

New stations

In this calibration 270 new stations were added to the post-processed forecasts (250 stations at 6-hourly and 20 stations at daily). Stations may be added to the post-processing if they are new stations provided by EFAS hydrological data providers or if their record of observed river discharge has exceeded the minimum requirement of 2 years. For EFAS v5.1, new stations have been added in 14 countries as detailed in Table 1.

Table 1: Number of new post-processed stations per country.

Calibration timeseries

The hydrological post-processing calibration requires at least 2 years of river discharge observations at 6-hourly or daily timesteps. Figure 1 shows the length of the record used to calibrate the hydrological hydrological post-processing models for each station. The exact dates used for the hydrological calibration for a station are shown in the 'EFAS calibration table' in the Post-processing tab in the reporting points (see EFAS Reporting Points). 

Figure 1: Length of the calibration record for each station. A minimum of 2-years is required for a station to be calibrated.

Calibration Methodology

The station models calibrated in the hydrological post-processing (HPP) calibration process contain the information necessary for the MCP component of the post-processing methodology. The MCP component is responsible for correcting errors due to the initial conditions and the hydrological model. A full description of the post-processing methodology can be found on the CEMS-Wiki here but for the purpose of keeping this page self-contained the following two key points should be noted:

1) The MCP creates a naïve "first guess" forecast for the next 15 days by conditioning the climatological distribution of the observed and simulated discharge on the recent observations and simulations values. This naïve forecast therefore only considers the autocorrelation of the timeseries and the recent state of the river, and does not include any information on the upcoming meteorological situation. The naïve forecast of the observations is referred to below as the Obs MCP. 

2) (After being spread-corrected) The EFAS ensemble forecasts are assimilated into these naïve forecasts to incorporate the meteorological information. This is done using a Kalman Filter so the resulting distribution is dependent on the distribution of the ensemble forecast, the distribution of the naïve forecast for the simulation (water balance), and the cross-covariance matrix between the water balance and the observations. The resulting forecast is referred to below as the Obs Full forecast.

Given these two points of the methodology we attempt to evaluate the Obs MCP forecast and the Obs Full forecast. The Obs MCP can be thought of as a lower threshold of the potential skill of the post-processed forecasts. Whilst there are situations where the skill for a specific forecast may be lower (e.g., ensemble forecast introduces a fictional flood event or an event is missed, erroneous observations are included in the process) the post-processed forecasts will in general be at least as good as or of higher skill than the Obs MCP. The Obs Full is calculated here using the water balance as the ensemble forecast. Within the post-processing method the ensemble forecast is considered a forecast of the simulation and not the observations (i.e., it's forecasting the water balance). Therefore, the perfect ensemble forecast would be the water balance itself (deterministic and accurate). As we are assimilating a "perfect forecast", the Obs Full is the post-processing equivalent of the hydrological model skill layer. Therefore, Obs Full can be considered an upper skill threshold although there may be occasions when the post-processed forecast performs better due to the interaction between the uncertainty of the ensemble forecast and the Obs MCP.

Calibration Results

In the following results the extremely poorly performing stations were removed and are being analysed separately.

Assessment of the CRPS skill score

The CRPSS is shown in Fig. 2 for the Obs MCP (blue) and Obs Full (orange) forecasts for six-hourly stations (upper panel) and daily stations (lower panel). The CRPS evaluates the full forecast distribution. The benchmark forecast is the water balance and the "truth" values are the observations. Over 95% of stations show an improvement compared to the water balance even when no meteorological information is included (Obs MCP). This is mainly due to bias correction. When the meteorological information is assimilated (Obs Full) all stations show an improvement with 67% of stations showing a reduction in error (in terms of the CRPS) of 50% even at lead-times of 15 days. 

 Figure 2: Continuous Ranked Probability Skill Score (CRPSS) for stations calibrated at six-hourly and daily timesteps.

The maximum day for which the CRPSS is above 0.5 for each station is shown in Fig. 3 for Obs MCP (upper panel) and Obs Full (lower panel). The skill for the operational post-processed forecasts is expected to be between these values. It can be seen that when the meteorological forcings are included (Obs Full) are included that the majority of stations are improved across the forecast period. 

Figure 3: Maximum lead-time (in days) when the CRPSS in above 0.5 for Obs MCP (upper panel) and Obs Full (lower panel).

Assessment of the KGE skill score

The modified KGESS is shown in Fig. 4 for the forecast median of the Obs MCP (blue) and Obs Full (orange) forecasts for six-hourly stations (upper panel) and daily stations (lower panel). Again, the benchmark is the water balance and the "truth" value are the observations. Improvement over the water balance is shown by 50% of stations at 10 days lead-time for the Obs MCP, however, over 25% of stations do show a degradation in the forecast median for lead-times greater than 3 days. This is unsurprising as no meteorological information is included and the information provided by the recent observations is limited at longer lead-times. When the meteorological information is assimilated (Obs Full) almost 75% of stations show an improvement in terms of the modified KGE at a lead-time of 15 days. The operational post-processed forecasts are likely to fall somewhere between the skill of the Obs MCP and the Obs Full forecasts depending on the accuracy and confidence of the ensemble forecast. There are some stations where the degradation of the modified KGE is severe. 

Figure 4: KGESS (benchmark is the water balance) for the forecast median of the Obs MCP (left/upper panel; blue boxes) and the Obs Full (right/lower panel; red boxes) forecasts.

Assessment of the KGE components

The following figures show the correlation, bias ratio, and variability ratio for the Obs MCP, Obs Full, and the simulation (water balance) to identify where the loss in KGE is occurring. All three components have a perfect score of 1. 

Figure 5 shows the correlation for the forecast median of the Obs MCP (blue) and Obs Full (orange) forecasts for six-hourly stations (upper panel) and daily stations (lower panel). When meteorological forcings are not considered (Obs MCP; blue) the correlation of the forecast median decreases from close to 1 to just below 0.5 for about 50% of stations. This is because the Obs MCP tends towards climatology at longer lead-times. For some stations, the forecast median shows a negative correlation with the observations at the longer lead-times but for other stations shows a near perfect correlation of 1 even at lead-times of 15 days. When meteorological forcings are considered (Obs Full; orange) the forecast median has a correlation above 0.5 for almost all stations at a lead-time of 15 days. 

Figure 5. Correlation between the observation and the Obs MCP (blue) and Obs Full (orange).

Figure 6 shows the bias ratio for the forecast median of the Obs MCP (blue) and Obs Full (orange) forecasts for six-hourly stations (upper panel) and daily stations (lower panel). When meteorological forcings are not considered (Obs MCP; blue) the bias ratio of the forecast median decreases from close to 1 to just above 0.8 for about 50% of stations. This is again because the Obs MCP tends towards climatology at longer lead-times. When meteorological forcings are considered (Obs Full; orange) the forecast median has a bias ratio above 0.9 for almost all stations at a lead-time of 15 days. 

Figure 6. Bias ratio between the observation and the Obs MCP (blue) and Obs Full (orange).

Figure 7 shows the variability ratio for the forecast median of the Obs MCP (blue) and Obs Full (orange) forecasts for six-hourly stations (upper panel) and daily stations (lower panel). When meteorological forcings are not considered (Obs MCP; blue) the variability ratio of the forecast median decreases from close to 1 to just above 0.8 for about 50% of stations. When meteorological forcings are considered (Obs Full; orange) the forecast median has a variability ratio above 0.8 for almost all stations at a lead-time of 15 days. However, these results do show that the hydrologically post-processed forecasts underestimate the variability of the river discharge.

Figure 7. Variability ratio between the observation and the Obs MCP (blue) and Obs Full (orange) .

Conclusions

Based on the results above, the Hydrological Post-Processing improves the behaviour of the EFAS simulations compared with the raw model outputs but care should be taken for extreme values that exceed those seen in the calibration period.