| Page info | ||||||
|---|---|---|---|---|---|---|
|
| Warning | ||
|---|---|---|
| ||
This page is under development and is subject to frequent changes. This is for internal use only. |
| Info | ||||
|---|---|---|---|---|
| ||||
|
| Info | ||
|---|---|---|
| ||
Access to ARCO data is programmatic; therefore, users of these resources and this documentation are expected to have some relevant programming experience. |
The Data Store Service (DSS) at ECMWF offers a subset of some of the data available in the Data Stores in an an Analysis Ready, Cloud Optimised (ARCO) format. These ARCO data are typically provided as Zarr archives stored in S3-compatible object storage.
...
Applications that require downloading large volumes of data for repeated, offline processing may be better suited to traditional data access methods via the relevant Data Store request service (e.g. cdsapi).
Analysis Ready
“Analysis ready” means that the data can be used directly in downstream applications without additional preprocessing.
Specifically, this means
...
that there is:
- no need to
...
- Decode decode packed variables
- Apply no need to apply scale factors or offsets
- Interpret no need to interpret non-standard or obscure metadata representations
- Merge no need to merge multiple files before use
This significantly reduces the workload for downstream applications.
...
Data in Zarr archives is stored in chunks. A chunk is the smallest unit of data that can be transferred. Partial chunk downloads are not possible. Therefore, the chunking strategy used has a major impact on performance.
...
| Note | ||
|---|---|---|
| ||
Your API key is available from your DSS profile page for one of the Data Stores you have registered with e.g. for the CDS If you have not already registered with the Data Store Service, you must register an ECMWF account and use this to log in to one of the Data Store portals, e.g. the CDS or the ADS. In addition to accepting the general DSS Terms and Conditions, you must also accept the licence associated with the dataset you are using from the relevant portal. Failure to accept the appropriate licence will result in authorisation errors. |
...
| Note | |||||||
|---|---|---|---|---|---|---|---|
| |||||||
You will need to install the following packages to access the zarr Zarr archives with xarray, they can be installed with PyPi or conda.
|
...
The following example is a xarray get-started guide, however this will not suffice for any "heavy duty" or operation operational workflows. Please see the Advanced Usage below for more robust workflows mechanisms.
...
Advanced Usage
The xarray interface to zarr Zarr does not offer any retry mechanism as default. Given the nature of this remote access to data it is quite possible that larger workflows may result in a failed transfer of a data chunk for a number of possible reasons, e.g. a temporary loss in connectivity.
To make your workflows more robust, you can include a retry mechanism as part of your connection to the zarr Zarr archives. Below are two examples using existing open-source libraries.
...