What is Hindcast Data ?
Hindcast data or 'Re-Forecasts' are the terms used to describe validated 'historical' data or 're-analysed' forecast data. The hindcasts are produced in the same way as the real-time forecasts but for past dates.
The main goal is to situate real-time forecasts on a probability distribution of many past real-time forecasts. This allows for:
- producing calibrated ensemble forecasts
- deriving forecast verification metrics
- assessing impact of changes to the forecasting system through time, e.g. resolution upgrades
- highlight significant forecasting events (extremes)
by building a climate dataset.
We re-run forecasts back in time (past 20 years) for a given forecast date with many ensemble members, e.g. 20 years x 11 members. Then we improve the signal-to-noise ratio by constructing the climate from many more reforecasts i.e 9 reforecasts for medium range, 5 reforecasts for sub-seasonal range.
How can Hindcast Data be accessed within ECMWF products ?
Access to hindcast data, is dependent on what type of hindcast data you would like.
If you would like ENS extended hindcast data, you should be able to access it with a real time licence. However, if it is seasonal hindcast data, you will need access to our archive via an archive licence/ account. You will also need access to the archive should you need historical hindcasts.
Max charge Real-Time users who pay for the Gold package will have access to the archive included under their service package.
You can browse the archive catalogue here: Archive Catalogue
And you can browse the real-time catalogue here: Real-Time Catalogue.
These are the sets and subsets of the Real-Time Catalogue from which you can obtain hindcast data:
- Group B: Ensemble forecast up to the sub-seasonal range:
- Set III: Atmospheric Model Ensemble 15-day forecast (ENS):
- III-i-a: Atmospheric fields - Single level
- III-i-b: Atmospheric fields - Pressure levels
- III-i-c: Atmospheric fields - Model levels
- III-i-d: Atmospheric fields - Potential vorticity levels
- III-i-e: Other surface levels, e.g. multi-level snow fields
- III-i-f: Human Thermal Comfort Indices
- Set VI: Atmospheric Model Sub-seasonal forecast:
- VI-iii-a: Weekly means of individual ensemble members - Single level
- VI-iii-b: Weekly means of individual ensemble members - Pressure levels
- VI-v-a: Individual forecast runs - 6-hourly - Single level
- VI-v-b: Individual forecast runs - 12-hourly - Pressure levels
- VI-v-c: Individual forecast runs - 6-hourly - wave
- Set III: Atmospheric Model Ensemble 15-day forecast (ENS):
If you are unsure whether your data requirements require a real-time or an archive licence, please raise a ticket at the ECMWF Support Portal.
Hindcast Data as a Real-Time Product
POST - 49r1: When we implemented cycle 49r1, hindcast data are produced every odd day of the month. So in practice, if your data request includes 'use=monday/thursday', this means that you will receive the hindcast data on Monday and Thursday, but only if that day is an odd day. To receive data every other day, it is better to have use=monday/tuesday/wednesday/... . Here you can find more info about the 49r1 cycle here: https://confluence.ecmwf.int/display/FCST/Implementation+of+IFS+Cycle+49r1 The dissemination of post 49r1 hindcasts is covered in more detail below. |
POST - 48r1: As per the link on Implementation+of+IFS+Cycle+48r1, configuration changes made under Cycle 48r1 offers two sets of re-forecasts / hindcasts: and one for the extended range:
Both re-forecasts runs twice per week from 00 UTC, on Monday and Thursday. Similarly under the wave model, Cycle 48r1 has two sets of re-forecasts:
There is no additional products from ENS-WAM Extended hindcast statistics and both re-forecasts, as before, run twice per week from 00 UTC, on Monday and Thursday. PRE-48r1: A four sets of dissemination files are generated by the Monthly Forecasting system each set containing the The re-forecasts or hindcasts are created twice a week (Mondays and Thursdays) and are ready a week in advance. Real-time extended range forecasts are actually calibrated using a 1-week window of re-forecasts, which represents a total of 660 (3 start dates x 20 years x 11 members) re-forecast integrations. For a Monday run, we calibrate using the date of that Monday plus the dates of the Thursdays immediately before and after. Likewise Thursday runs use the date of that Thursday and the two Mondays either side. Users have a choice to distinguish between daily and monthly ENS products on the requirement level. The reference date of the hindcast (date of the real-time forecast associated to the hindcast) REFDATE=YYYYMMDD is set by default in dissemination to the latest operational Monthly Forecast cycle date and does not have to be specified in the requirements. Also, for hindcast data, the only value for the parameter DATE (date on which the hindcast is based) supported in dissemination is ALL. The raw hindcast monthly forecasting data are a 4-member ensemble. The wave model monthly forecasts are stored in the stream waef for the real-time forecast and in the stream enwh for the hindcast. |
We do not distinguish between forecast and reforecast in MARS storage, so the hindcast data is stored here: Archive Catalogue
You should see the hindcasts further back in time, but please note this data will not be disseminated as it is generated far in advance and is static. That said valid data (current data) cannot be 'valid' via the MARS, so some MARS user requests are rejected because it may include Valid Data, which is restricted via an archive licence.
We define 'Valid data' as data for which the value of DATE + TIME + STEP (or FCMONTH) + 24 hours is greater than or equal to the current date, i.e. any data with a verification time in the future or less than 24 hours in the past. You can also find more information on calculating Valid Data here (under point 12): Guidelines for Interpretation of the Rules and Standard Licence Agreement
Gold & Silver Users: Installing Hindcasts In PREd
A user with a Gold or Silver service package, you may have access to PREd. You can then make changes to your data requirements via your data streams and submit publication requests in PREd. Instructions on how to modify streams and add data in the PREd can be found here: Installing data using the PREd. However, please consider the below examples if you would like to install & receive hindcast data.
I would like to install ensemble forecast hindcast data on single levels for Saturday and Sunday for the UK for the parameters 10U, 10V, 2T with base times of 00z and 12z (7-days) and 12-hourly time steps. What dissemination code do I need to use in PREd? Firstly, you can use the snippets button (<>) to find: # Group: Group B: Ensemble forecast up to the sub-seasonal range # Set: III: Atmospheric Model Ensemble 15-day forecast (ENS) # Subset: III-i-a: Atmospheric fields - Single level (pf) You will need to change the area according to your own data needs. In this example, those for the UK.
"type=pf" refers to the perturbed forecast from the ensemble forecast.
"type=cf" refers to the control forecast from the ensemble forecast.
"stream=enfh" refers to ensemble forecast hindcast for probabilistic forecasts. "use=saturday/sunday" refers to the days for which you would like the hindcast forecast from the ensemble.
Again, "stream=enfh" refers to ensemble forecast hindcast for probabilistic forecasts. "use=saturday/sunday" refers to the days for which you would like the hindcast forecast from the ensemble. |
I would like to install ENS-EXTENDED global hindcast data on single levels for all days of the week for the parameters 2T and TP with base time of 00z and 6-hourly time steps. What dissemination code do I need to use in PREd? Firstly, you can use the snippets button (<>) to find: # Group: Group B: Ensemble forecast up to the sub-seasonal range # Set: VI: Atmospheric Model Sub-seasonal Forecast # Subset: VI-iii-a: Atmospheric fields - Single level You will need to change the area according to your own data needs.
"type=pf" refers to the perturbed forecast from the ensemble forecast. "type=cf" refers to the control forecast from the ensemble forecast. "stream=eefh" refers to ensemble extended forecast hindcast for probabilistic forecasts. "use=monday/tuesday/wednesday/thursday/friday/saturday/sunday" refers to the days for which you would like the hindcast forecast from the ensemble. "type=cf" refers to the control forecast from the ensemble forecast. |
I would like to install ENS-EXTENDED Wave hindcast data on single levels for Monday and Tuesday for the parameter SWH with a base time of 00z and 6-hourly time steps. What dissemination code do I need to use in PREd? Firstly, you can use the snippets button (<>) to find: # Group: Group B: Ensemble forecast up to the sub-seasonal range # Set: VI: Atmospheric Model Sub-seasonal Forecast # Subset: VI-v-c: Individual forecast runs - 6-hourly - wave You will need to change the area according to your own data needs.
"type=pf" refers to the perturbed forecast from the ensemble forecast. "type=cf" refers to the control forecast from the ensemble forecast. "stream=weeh" refers to wave ensemble extended hindcasts "use=monday/tuesday" refers to the days for which you would like the hindcast forecast from the ensemble. |
Disseminating Hindcast Data to our users (Post 49r1 Cycle)
Data will be disseminated sufficiently far enough ahead for you to build a model climate with the following configuration.
The medium range model climate can be built from 9 re-forecasts (1-central + 4 ahead + 4 behind) annotated by the red dashed line.
For sub-seasonal re-forecasts, the model climate can be built from 5 re-forecasts (1 central + 2 ahead + 2 behind) - annotated by the red dashed line.
Our strategy is to run reforecasts ahead of time to allow users to collect the large amount of data needed consistently. This means if today is 26th June, for example, all the required reforecast data must have reached users at 21:00z at the very latest on 21st June for medium-range and 23rd June for sub-seasonal.
Please refer to the pages below for a more in-depth description on the current configurations:
Sub-seasonal range re-forecasts configurations changes - Forecast User - ECMWF Confluence Wiki
Medium range re-forecasts configurations changes - Forecast User - ECMWF Confluence Wiki
Terminology Summary
- A Reforecast: a full 20-year set or model initialisations to simulate past conditions of today's forecast, so 20 years x 11 ensemble members
- Calendar date: itself
- Schedule date: The date when ECMWF publishes its hindcast and reaches the user
- Production date: The date the reforecast is produced in ECMWF
- Product generation date: The date when the Pgen software uses the user data request to produce the output products we need to disseminate to the users.
- Reforecast dates: The start dates of the re-initialisation of the IFS model for a specific forecast date e.g. 20040625/20050625...20250625
FAQs
| General |
|---|
Q: Can you confirm how far back the hindcast data goes? A: Normally, re-forecast data is produced for a given calendar date for the past 20 years. But if you are after specific archive / historical hindcasts, please browse our archive catalogue. |
Q: I am a gold service package user. Does our license support getting hindcast data? And what is our options of getting the data and what are the methods of getting the hindcast data? Since you are a user with a Gold service package, you can make these these requirement changes easily in PREd. Instructions on how to modify streams and add data in the PREd can be found here: https://confluence.ecmwf.int/display/DAC/Installing+data+using+the+PREd To add hindcast data please follow the instructions via the link and use the snippets function. Please search for 'hindcast' which will bring up examples of both the ENS and ENS Extended hindcasts. Once you have your hindcast data configured, please launch a publication request. See above for examples of installing hindcast data. Please note, if the addition of the hindcast data brings you above your Volume Band we will be in contact about amending your current contract. Your gold service package also allows for access to the MARS archive and hence any historical hindcast data you may wish to use too. |
| MARS Archive |
Q: I have a MARS archive licence with ECMWF. Why when I try and submit a request for hindcasts I get an error message: do I not have access ? A: Please raise a ticket at the ECMWF Support Portal and send through a copy of your request. It may be possible you are requesting valid data rather than archive data or other data which is not included under your MARS archive licence. |
Q: I am looking for is historical forecasts with origination date of a given month that forecast 7 months ahead. Do you have forecasts of temperature and precipitation from Oct 2024, Oct 2023 for example? A: You would need a MARS archive licence for this request. From here https://apps.ecmwf.int/archive-catalogue/?class=od you can select the "Seasonal forecast" category, and then explore the years/months, the levtypes and then for every of them the available parameters. |
| Real - Time |
Q: What types of perturbations are used to create the 11 ensemble members? More specifically, are parameters perturbed always perturbed in the same way for a given ensemble member? Is there a special configuration for ensemble member 0, compared to the other members? A: The reforecast ensemble is constructed with initial perturbations based on a combination of Singular Vectors ensemble of data assimilation (EDA) members from ERA5. During the integration model perturbations are added with the Stochasticallt perturbed parameterisation tendency (SPPT) scheme. This works in the similar way to the real time ensemble. Member 0 is the unperturbed member. |
Q: If it is a leap year, what should we expect for initialization dates for Feb 29 hindcast dataset? A: Normally, re-forecast data is produced for a given calendar date for the past 20 years. This is not possible for a leap day. The re-forecast data will therefore indeed be based on the 28 Feb for the past 20 years. The re-forecast date (hdate) of the grib fields will be 28 Feb. The date of the disseminated file and of the grib fields will be 29 Feb. |
Q: Are there any plans to include hindcast data for either the IFS 15-day ensemble forecast or EC46 in the open data initiative? A: At the moment, we don't plan to do this. We're trying to control volumes and establish the most appropriate dataset that is sustainable, meets WMO requirements, etc. The S2S dataset has some hindcast data that is available out to 46 days. |
Q: What is the centre of the climate for today? A: On even days, it is not possible to use a centralised re-forecast so we recommend using the closest re-forecast as the 'central' and then 4 either side.
With a wider distribution of the re-forecast, the next odd day must also use a previous re-forecast as the 'central' with 4 either side.
After a re-forecast/model climate has been applied 4 times, on the next even day we need to use the next re-forecast day. We started from the 17th, but the same date is also use for the 16th. Same as on 20th we use the 21st, on 16th we us the 17th as the central date.
After this, the next re-forecast becomes the ‘central’ position. For example, the 16/17/18/19 of the month will use 17, but 20/21/22/23 will move to 21.
In essence, you can use the same model re-forecast for 4 days in a row EXCEPT at the end of the month. |
Q: Which reforecasts would we need for today?
So, if we were on the 26th for example, the central climate is 25th, so you will need to construct the climate with 9 reforecasts:
|
Q: What data will we get today? A; If today is 26th, shift the date by 5 (or by 3) reforecast dates and on the 21st, to allow for 4 days preparation, you will need to have received the 9th (furthest reforecast date of the climate). |
Q: If we make a change to our request today, when will we first see that change in the data? A: Shift the date by 1 + 5 (or 3) reforecast dates. So, on the 21st, to give 4 days to users to prepare, and for ECMWF to prepare, we need to have produced the 9th (furthest date of the climate) one hindcast earlier than it is disseminated. |
Q: If we make a change to their request today, when will we be able to construct a full climate with that change? A: 9 + 2 reforecasts dates later. |
