EFAS v5.0 - Flash Flood Skill Assessment

This analysis evaluated the skill of the ERIC flash flood products when compared against flash flood observations, the results are used to decide the criteria for issuing flash flood notifications. It was necessary to perform a new skill assessment for EFAS v5.0 to decide these criteria, rather than using the criteria from EFAS v4.1 because of the following changes:

• The new calibration of the LISFLOOD hydrological model which is used within the generation of the ERIC flash flood products
• The ERIC products are now calculated directly from the surface runoff predictions from LISFLOOD, previously precipitation was combined with LISFLOOD predictions of soil moisture to estimate surface runoff

The evaluation was performed for ERIC flash flood predictions between 1st January 2022 - 31st December 2022

Flash Flood Observations

Observations of flash floods during the evaluation period were obtained from three sources:

• EFAS partner feedback
• FloodList.com
• EFAS long term run (river discharge re-analysis)

EFAS Partner Feedback

EFAS partners are able to provide feedback in response to each flash flood notification email they receive, they can inform if the flash flood event did or did not occur. Therefore the feedback received during the evaluation period was selected and events where a flash flood did occur were chosen. 42 flash flood events were reported by EFAS partners during the evaluation period

FloodList.com

Flash flood events were obtained from an API collated by FloodList.com. The API was queried to return events which matched the following criteria:

• Occurred during the evaluation period
• Occurred within the EFAS domain
• Flood event type was recorded as either 'flash flood' or 'minor flood', the latter was included as this refers to flooding with a small spatial and temporal scale which is similar to flash flooding

Some of the extracted flash flood events were recorded as having a duration of several days, whereas normally flash flooding occurs within a 24 hour timescale. This longer duration could be because other types of flooding were recorded during the same event, for example longer lasting riverine flooding may have occurred later on further downstream. Uncertainty in the timing of the reported flash flood event could also explain the longer duration. In these cases it was decided to only record the first date on which the flash flooding was reported as this would coincide with the initial heavy rainfall which is often associated with most flash flooding impacts.

  EFAS Long Term Run Re-Analysis

Flash flood observations were also derived from the EFAS historical long term run, which is generated by forcing the LISFLOOD hydrological model with gridded 6-hourly meteorological observations. Jesus Casado at the JRC has already used this dataset to derive flood events at gauging stations provided by EFAS partners with an upstream area >= 500 km², where the river discharge exceeded the 5 year return period threshold. The same methodology was used here to identify flood events at gauging stations with an upstream area between 500 - 2000 km², the latter is the maximum upstream area for which flash flood notifications are issued. From these extracted flood events, flash flood events were defined as events where the duration of the 5 year return period exceedance was no greater than 1 day. Events with a longer duration were discarded as they were considered to be longer lasting riverine flood events. It should be noted that EFAS version 4 of the historical long term run was used rather than EFAS version 5, this was due to there being insufficient time to perform full quality checks on the EFAS version 5 data.

Combined Observations

Flash flood observations from each of the above three sources were combined into a single dataset. Events were aggregated onto the same administration regions which are used to issue flash flood notifications. If multiple events were reported in the same administration region on the same day, only one flash flood event was recorded, this reflects that in the same circumstances only one flash flood notification would be issued.

During the evaluation period, the majority of the flash events occur towards the end of 2022 (Fig. 1). The spatial distribution of the reported flash flood events shows that the greatest number of events occurred around the Mediterranean, including Italy and the Balkans (Fig. 2).

ERIC Flash Flood Forecasts

Flash flood forecasts were generated at 00 UTC for each day during the evaluation period from the 1st January to the 31st December 2022. For each forecast, the LISFLOOD hydrological model was forced with the 20 member COSMO-LEPS meteorological forecast, the surface runoff output from the hydrological model was then used to calculate the ERIC surface runoff indicator and generate the forecast products shown on the EFAS website. 

Evaluation Methodology

ERIC forecasts produced at 00 UTC on each day during the evaluation period were evaluated. A range of different exceedance probability threshold values of the 2, 5 and 20 year return periods were tested ranging from 0 to 100% in increments of 10%. The evaluation was performed separately for lead times of 0-24h, 24-48h, 48-72h, 72-96h and 96-120h.

For each forecast, the reporting points were extracted, then each of the different exceedance probability and lead time thresholds were applied. For each application of the different thresholds, the ERIC reporting points which satisfied the threshold criteria were extracted. For each extracted reporting point, the date of the forecasted flash flood event and the ID value of the EFAS administration regions layer were extracted. These were then compared with the observations to see if there was a corresponding observation in the same region on the same day (hit), or if there was no corresponding observation (false alarm), or if there was an observation but not forecasted event (miss). This process was repeated for every forecast during the evaluation period and the total number of hits, misses, false alarms and correct negatives was calculated.

The skill of the forecast for each unique combination of exceedance probability and lead time threshold was calculated from the corresponding hits, misses and false alarms using the f(beta) score, which combines the recall (also known as the hit rate) and precision (also known as the false alarm ratio) abilities of the forecast. This score was also used by Jesus Casado from the JRC in their evaluation of the EFAS formal and informal notifications.

f(beta) = (1+ beta2) * Hits / (1+ beta²) * Hits + beta² * Misses * False Alarms  

beta = a predefined parameter which by default is 1.0 giving equal weight to recall and precision. A sensitivity assessment is performed with different values of this parameter.

Evaluation Results

The results below show the f(beta) score for the different exceedance probabilities of the 2, 5 and 20 year return periods. The highest scores occurred for a 30-40% exceedance probability of the 2 year return period at 0-24 and 24-48 hours lead time, and also a 10-20% exceedance probability of the 5 year return period at the same two lead times. The results from the exceedance of the 20 year return period had lower f(beta) scores, this could be due to fewer forecasts exceeding this threshold which results in a great number of missed events.

Sensitivity Analysis Beta Parameter

The beta parameter used in the f(beta) score controls the weight given to the components of recall and precision. A beta value lower than 1.0 means more weight is given to precision which means more a greater emphasis on reducing false alarms. Feedback from EFAS partners has suggested that the large number of false alarms received from the flash flood notifications is a concern for them. Therefore, a sensitivity analysis was conducted where different beta values <1.0 were used to compute the f(beta) score.