...
The CDS strives to deliver data as fast as possible, however, it is not an operational service and should not be relied upon to deliver data in real-time as it is produced.
Here we will try to give some context of on why requests can take time:
Data for the CEMS-Floods (EFAS and GloFAS) datasets are held within MARS at ECMWF.
The MARS Service service is a system designed for the request of GRIB Files files based on a Disk Cache disk cache and Tape tape storage architecture.
Most The most recent data is held on disk cache with , while all data being available from Tape.
available data is stored on tape. When a user requests data, the CDS queues the request based on the CDS own queueing priorities using the factors described above.places that request in a queue. Requests are prioritized by the CDS based on the factors listed above.
Once the request becomes eligible, Once the job becomes eligible it is passed to the MARS Service service at ECMWF for extraction of the relevant fields. It is only at this point that you will see your job request as 'Running'
Selecting areas When a user selects an area of data, it does not mean that you are not retrieving the whole globeentire global dataset. Each timestep of each date of each variable is classed as an individual grib GRIB field. MARS extracts sub-areas of data by retrieving the entire global grid and , cropping the selected area and , then returning the requested area to the user.
MARS as is a separate service to the CDS which also has constraints on its workload and . MARS has separate its own QOS limits that apply applicable to jobs for data requests, as it is a service shared across Operational services 'ie operational services (i.e. producing ERA5 and , GloFAS, and non-operational services such as the CDS).
The CDS service, from time to time, can experience periods of high user activity and increasing queue time for even small requests. In During these times we ask you to kindly wait for the queue to be processed, as there are fixed slots available that cannot be increased.
Figure 1 shows a period of high user activity. GloFAS and EFAS products are served by the adaptor.mars.external service, you can see that the active users (blue line) is are well above the green line of 50 slots allocated to the GLoFAS and EFAS requests (green line). When the blue line falls again below the green line then the total queued users start decreasing until eventually there is no queuing time for any user request.
...
It is also very important that, when only a part of the geographical domain is needed, the user crops the data to a region of interest (ROI). This will help keep the downloaded data size as small as possible.
In Table 2, we list the maximum fields per request that you can retrieve for each dataset and the corresponding downloaded data size, assuming that you are:
- requesting GRIB2 file format
- no not cropping
- requesting the shorter time steps (6 hours vs 24 hours), when available
- requesting ensemble perturbed forecasts
...
Indeed the CDS system penalises users that submit too many requests, decreasing the priority of their requests. In short:Too many parallel requests will eventually result in a slower overall download time time.
For this reason, we suggest limiting to a maximum of 10 parallel requests.
...