Request submitted by the user and arrives in the Web API queue
The request arrives to the service back-end and a new record is added in the service database. (MongoDB).
- The request is not yet visible on the MARS activity
Each request is stored in MongoDB. The corresponding record includes all the request's details (status, timers, host, user, etc).
The request record is updated periodically to reflect its current status.
The "queue time" depends on several factors, ie priorities, limitations, current load, etc
Please note that to improve the service some QoS limitations and priorities are applied, like:
The total maximum number of active requests
The maximum number of active requests per user
- The maximum number of queued requests per user
Once an API slot is available and if all the QoS rules are met, the request will become "active" on the Web API level.
The request becomes active on Web API level
Split: On the Web-AP, I level, the request is split into a number of smaller requests to achieve better performance from the MARS point of view.
For instance, if a request is requesting one year of "ERA-interim" daily data it is split per month. ie it is split into 12 requests.
Create the file: Once all the sub-requests have completed successfully the corresponding sub-files are merged into a final one and the request status becomes complete
If something goes wrong during the procedure above the request status will get the value "aborted"
The request becomes active on MARS level
At this phase, the request becomes visible on the MARS activity (with status active or queued in MARS)
Note that a request can be at status active on the level of API but queued on MARS.
We always try to tune the Web API and MARS systems in order to improve the performance but it most cases it is not so straightforward.
On the level of MARS server, there are several MARS QoS rules (ie limitations, restrictions and priorities) like the ones below.
On marsod-core, the total number of Requests from ECMWF Web API with 1 tape mount [source] is limited to 6
On marsth-core, the total number of Requests from ECMWF Web API with 1 tape mount [source] is limited to 12
To avoid long queuing times it is strongly recommended to improve your Retrieval efficiency
File transfer procedure
The produced file is then transferred by the Web API client from the ECMWF to the user's local disk.
Once the file has been transferred successfully the file will be removed from our disks and it won't be any more available.
If the transfer procedure fails
The file transfer procedure may fail due to several reasons (eg network connectivity problems between our end and the users)
- If the transfer procedure fails difficult to trace back what has happened
The API client checks the volume of the transferred file and if the size is not the expected one the transfer procedure will start again. The MAX number of retries is 10
- If the client is not successful, the user should be able to manually download it from the Job List.