- This web page provides a summary of points raised and questions from the User Voice Corner breakout sessions from the UEF meeting held in June 2023 at ECMWF. These are in italics.
- The ECMWF response at the time is then shown in standard black font (no italics).
- Then a current 'status report' is provided in red text, for certain topics, to provide further information and/or to show how and where certain topics have progressed since that meeting (as of 19 April 2024).
There were 4 main topic areas covered in the breakout groups - as denoted by the bold headings below.
Technical
How can users process ENS data given the increasing volumes ?
- Various options for users exist, from both software and hardware angles. ; computing products there from voluminous model data before disseminating just those products downstream. Another option is to make more/better use of data compression for any GRIB2 fields. Finally one could also employ bigger pipes for data transfers (though admittedly that is not a practical option for everyone). Meanwhile ECMWF is currently investigating network changes to address some overall transfer speed degradation.
What is the timeline for the newly agreed WMO Dataset ?
- From 29 February 2024 ECMWF increased the resolution of open data to 0.25 degrees.
Is there a way to create subregions of data and download only those from MARS for non-MS/CS users ?
Can a user migrate fully to grib2 before cycle 50r1 (at least for the parameters they receive) ?
- This is not clear - it would need extensive testing.
Can ECMWF split dissemination files as this would help with data volumes with increasing ENS resolution ?
- This has been considered internally since UEF. Potentially we can do this (and indeed we do do this for our own needs), but to be able to implement anything we would need to know what type of splitting users have in mind. There are a number of options. We would encourage interested parties to raise an 'issue' on the ECMWF Support Portal using this link.
User highlighted that the next cycle test candidate phase (49r1) would be too short for training/calibration.
- There are many constraints for ECMWF in this regard. The Member State Release Candidate Phase (RCP) for cycle 49r1 is planned to begin in late June 2024, with implementation then planned for Oct/Nov 2024. During RCP, we expect to have all 49r1 test data available to users in near-real-time through test dissemination streams. Some test data should be available a month or so before RCP through MARS.
Could there be a single place to download common esuite/oper data easily, as MARS is slow and complicated ?
- Dissemination is the main alternative, offered by ECMWF, to MARS downloads. This is less complicated! And as stated just above dissemination of esuite output will be available during the RCP.
Machine Learning
Could ECMWF provide more advice on best practices for downloading data for machine learning applications ?
- There are guidelines to retrieve data efficiently here: Guidelines to write efficient MARS requests. If there are particular datasets needed for Machine Learning applications the User Services and MARS Team could provide specific guidelines. The requests for those datasets would help us (and we could try to optimise access to them).
Users would like more information about our forecast/hindcast/reanalysis products wrt their advantages and disadvantages for different ML applications.
Comments related to possible future use of forecasts based on machine learning:
“Users do not care about how forecasts are produced ”, i.e non-meteorologists might not be too bothered about that type of model that is behind the forecast, and also a NWP model is already a black box for them.
- This is fair, but many other users have experience of strong and weak points of classical models, and use that knowledge to good effect.
There is a need to build trust and confidence in the ability of ML forecasts to predict rare events.
- This is work-in-progress. Many groups around the world are examining such aspects. Remember that resolution is one limiting factor that does not go away just because the model is an ML model.
In the private sector they do not seem too worried about trusting the model as long as it gives them an accurate forecast for that day.
It is important to evaluate the meteorological consistency in ML forecasts (not only scores)
- Again this is an ongoing area of study, in and outside of ECMWF. Although ML models learn from reanalyses like ERA5, and inherit some ERA5 characteristics, the physical consistency inherent in ERA5 is not ensured in an ML model. Since UEF 2023 ECMWF has produced a lot of ML model evaluation (e.g. see our AIFS blogs here, and also the preprint here).
How can we best use all the experience that forecasters have built up from using NWP model when moving to ML ?
- Remember ML models have learnt from an old IFS system (ERA5), so will have some similar characteristics, albeit with some of biases - e.g. for tropical cyclone movement - diminished
The participants also wanted to see ECMWF evaluation for ML models, and the use of our expertise to understand them.
- This is a fast growing area. We have shown that on average, for broad scale patterns, the latest AIFS version beats HRES by a small margin. There are also some hints of AIFS output being less "jumpy".
ML models could also be used to detect weak points and limitations in NWP and either try to solve them or replace them with ML. A complementary use of ML models could also help to explain outcomes from NWP, e.g by running fast additional sensitivity experiments. Similarly, there is an appetite for using explainable AI methods but perhaps a better understanding of these methods and their limitations is needed.
For climate modelling, UKMO is exploring how to use a ML model as a downscaler of physical based climate simulations.
Machine learning could also help to achieve seamless forecasting by improving the "stitching" process.
Extended Range and Seasonal
Q. How can the 51 member ENS and 101 member EXT best be combined to provide a seamless forecast to users?
ECMWF does not have an immediate answer to this question.
- A new recruit started at ECMWF in February 2024, and he is looking directly at the two ensembles alongside one other (from 00Z DTs) with a view to blending their outputs together, in various ways, to provide even better forecasts. This could also feed into producing a more seamless product across the D15-16 join.
Q. What will be the temporal frequency of output in the early stages of the EXT forecasts? Will it be hourly as is presently the case in the combined ENS/EXT ensemble?
- No, the output of the EXT forecasts is 6 hourly throughout the forecast range, at least for Cy48r1. It was commented that for many users this would severely limit the ability to combine 51 member ENS and 101 member EXT to produce a larger ensemble over the first 15 days (see ECMWF comment to the question immediately above). Frequency of output was particularly important for wind, even sub-hourly is important. ECMWF noted that they respond to user demand; if people wanted this data they should request it, provision can then be considered against technical constraints and costs.
Q. Will the changes in EXT lead to any increase in skill for weeks 5 and 6?
- Skill at these ranges is challenging. Nonetheless, evaluation at ECMWF has shown some marginal benefits from larger ensemble sizes, which the new EXT configuration will supply (both directly, and via the possibility of lagged averaging e.g. yesterday and today's forecast at longer ranges). We are also striving to continually improve the accuracy of the forecast model and the range of processes it includes, which might also give incremental skill improvements at longer range.
- When using extended or seasonal forecasts ECMWF strongly encourages users to directly reference skill metrics so that any signals seen can be correctly placed into context. For example a user could and should ask - "if I see a strong signal in (say) week 4 is it likely to be statistically reliable?" To see various skill metrics in open charts, click on the "Verification" facet check box on the left side of the entry page.
Q. Could ECMWF provide flexible visualisation for different averaging periods, particularly for the extended range?
- This is potentially a heavy task. With the new EXT configuration, weekly means will be calculated both at fixed lead-times and for fixed verification weeks, resulting in a large number of different weekly means for each forecast. Plots are only being produced for standard calendar weeks, though, to keep plot volumes manageable. Users always have the option of working with the daily data, although it is admitted that this can also be relatively heavy. Another example given was to be able to compare a monthly mean from EXT with the corresponding monthly mean from SEAS.
Q. Do weekly mean anomalies of Tmax and Tmin exist, since Tmean is often not so useful?
- Weekly mean anomalies of daily Tmax and Tmin do exist in MARS. We may consider to include these parameters in ecCharts if users think it could be useful.
Q. Given the increased ensemble sizes, will the MARS limits for number of fields in a single download be increased? Will the layout of data on tapes change?
- MARS limits exist to help the data handling system run as efficiently and smoothly as possible, and depend on many factors including physical capacity of the systems.
- Limits are kept under review in order to improve access. We advise users to follow general guidelines (Guidelines to write efficient MARS requests), so that they do not need to adjust their retrievals after changes to the forecasting systems, and so that requests work efficiently independently of the dates they want to retrieve.
- With regard to the resolution of the medium range ENS, that doesn't affect the number of fields per request, but the volume.
- With regard to the increased number of ensemble members in extended range forecasts, no, we have not increased the limit on the number of fields per request, as we would need to review/decrease again when the resolution increases.
- If there are retrievals for specific datasets where users believe a change would bring improvements, we would be happy to look into it.
- Note that the layout of data on tapes has probably changed due to the increase in size of individual hypercubes.
Q. Are there any changes in graphical products?
- A few old and obsolete products were dropped (stamp maps, old clustering) - documented here (see Web Charts section).
- ECMWF is currently investigating how to improve the some of the standard "weekly mean anomaly" charts, in order to convey a wider and more helpful range of basic information to users. This will reference not only the mean anomaly forecast but also spread. We may also deprecate the WMW statistical significance test we currently use which has limitations. ECMWF may present some new ideas in this area at UEF 2025.
Q. Noting that "SEAS5.1" was made available to users in gradual fashion, will SEAS6 re-forecasts be produced "on the fly"?
- No. SEAS6 reforecasts will be produced in advance of the switch of operational systems, as is/was the case with SEAS5.
- SEAS5.1 is the ECMWF contribution to C3S. It is generally labelled "system 51". This consists of a 1-degree interpolation for a subset of the SEAS5 variables, offered under the ECMWF open data policy. This dataset was created in 2022. The interpolation is different from that published by C3S as 'system 5' between 2017 and 2022; the update was needed because of software availability at the time of migration of HPC infrastructure (from Reading to the new ATOS in Bologna).
The release of the system 51 reforecast data was gradual, to reconcile the requirement for publication of reforecasts in time for use with the corresponding real-time forecasts with the resources available for the generation of this dataset. 'On-the-fly' usually describes data which is re-created each time it is published (like the reforecasts for extended range); this was not the case here - instead a fixed dataset was released, bit by bit, as it was produced.
users with direct access to MARS (MSs, commercial customers), who use data at the original model resolution, saw no change to their data at the point of HPC migration.
this change and its implications for users are in CDS documentation, as per the announcement of 12 October 2022 (to be found under the 'Announcements' heading of the documentation tab of the seasonal forecast catalogue entries; direct link: https://confluence.ecmwf.int/display/CKB/Announcements)
Precipitation
Q: can we have more types of CAPE - e.g. downdraught CAPE and slantwise CAPE - we are interested in anticipating strong gusts, that can relate, for aviation purposes?
- ECMWF has expanded its range of CAPE-related indices in recent years, and has been and is deprecating old ones which were less scientifically valid. This has all been in response to user requests. We have tried to focus on the main needs here because there is a limit to how many new variables we can routinely archive. In addition, ECMWF continues to work with ESSL (the European Severe Storms Laboratory) to try to improve forecast of convection-related hazards, such as very strong gusts.
- A Jupyter Notebook is now seen as the most expedient solution to meet user needs for other instability indices. ESSL have provided ECMWF with such a notebook for computing various convective indices such as Storm Relative Helicity (SRH), various CAPE indices incl. surface-based CAPE (SBCAPE), Lifted Index (LI) etc. A current problem regarding publishing this Jupyter notebook for users is that it quickly runs out of memory, so computation optimisation is required. We hope to be able to complete this optimisation by the end of 2024, although priority could potentially be raised given sufficient user demand.
Q: Why is ECMWF over-forecasting snow on the south side of the Alps ? LAMs can do the same; maybe this relates to boundary condition (BC) "contamination" as those BCs often come from ECMWF?
- ECMWF would like to know whether the upgrade in the ENS resolution, from 18km to 9km, that was implemented with cycle 48r1 in June 2023, has delivered any tangible improvements. It should have helped, although we know there are still various snow-related issues in the IFS (e.g. see here). For problematic cases, and for subsequent investigations please pass dates on to ECMWF, along with problematic forecast lead times.
Q: Please can we have more percentiles, from the ENS, in ecCharts - e.g. for CAPE
- With cycle 48r1, we switched to pre-compute ENS percentiles as on-demand computation was not feasible any more. That improved the performance of ENS percentiles layers in ecCharts. Currently, we provide ENS percentiles of 14 parameters (additionally total, convective and stratiform precipitation are all provided in 6, 12, 24 hour intervals). The next step is to add more parameters including CAPE; we plan to do that in 2024.
Q: In Korea we have to do systematic bias correction of precipitation totals. Can ECMWF help?
- We have created a bias-corrected post-processed rainfall product, from the IFS, called "ecPoint" (rainfall). Currently some ecPoint output is in ecCharts and OpenCharts. The actual (grid-scale) bias correction applied is situation-dependant (e.g. varying from about *0.3 to *2.0). The percentile values that users can access in ecCharts incorporate this bias correction, though currently the bias-corrected gridscale values themselves are not available as such.
Q: One user noted that the IFS over-predicts the occurrence of light rainfall, and similarly can deliver some rainfall from shallow cumulus clouds when other models do not.
- We are aware of an over-prediction frequency bias of light rainfall and an under-prediction of rainfall in heavier precipitation events (again ecPoint can help here). Although average rainfall totals did reduce slightly in cycle 48r1, there continues to be an over-prediction of small totals in certain situations; this is a topic of ongoing ECMWF research and we expect to mitigate this in cycle 50r1 (2025).
Q: In SE France the Foehn effect modulates rainfall totals, but this impact is underestimated by the IFS.
- Cycle 48r1, which went live in June 2023, has finer spatial resolution and better orography. This should have led to better, if still imperfect, rainfall distributions.
Q: Some users are unhappy with ECMWF's convective parametrisation, particularly for handling heavy convective rainfall.
- We acknowledge that there are some issues with convective rainfall, as in many global models, and that is one reason for km-scale resolutions which can often better represent the heavy precipitation in more extreme convective events. One other issue for convective parametrizations is the lack of inland advection of convective cells, particularly for inland snowfall from oceanic convection. There was a minor improvement in this regard in cycle 48r1 and it continues to be a topic of ongoing research at ECMWF, to improve the inland penetration of convective rain/snow.
Q: Another user highlighted unrealistic convective rainbands.
- These are also known about, but this is proving to be a difficult problem to solve.