Introduction
A new state-of-the-art infrastructure, referred to as the Common Data Store (CDS) Engine, has been developed by ECMWF to host the Climate and Atmospheric Data Stores. All layers of the infrastructure are being modernised: the front-end web interface, the back-end software engine, and the underlying cloud infrastructure hosting the service and core data repositories.
The modernisation of the Data Stores is a journey on which we embarked for the users and with the users. Without your support and your collaboration, this transition would simply not be achievable. We thank you all again for your patience during this significant modernisation of our Data Stores and eagerly await for your constructive feedback via our Support Portal!
What does this mean in practice to current CDS (and ADS) users?
User account:
ECMWF Single Sign-On (SSO) has been extended to all its platforms including the new CDS, ADS, and EW-DS. SSO allows users to have access to multiple applications by signing in with only one account. It also has other important features in regard to simple management, security, ease of use, and seamlessness.
All users of the new CDS, ADS, and EW-DS will need an ECMWF account. If you already have one, great. If you do not yet have an ECMWF account, please register with ECMWF to get one.
As a consequence of the single sign-on implementation, your current CDS or ADS login credentials will not work on the new CDS, ADS, EW-DS infrastructure, regardless of whether you are currently using the CDS or ADS interactively (e.g. over the web interface) or programmatically (using the API service). A new UserID (ECMWF) and API key will be needed.
Your ECMWF account will only allow you to login to the new CDS/ADS/EW-DS, not onto the current CDS/ADS.
You cannot use your ECMWF account to log in onto the current CDS or current ADS. SSO does not apply to current CDS nor current ADS.
Your existing CDS account will only work in the current CDS, not in EW-DS Beta. You need an ECMWF account to start using EW-DS-Beta.
CDS/ADS/EW-DS Catalogue - data download over web interface:
- Before downloading any data from the new CDS/ADS/EW-DS infrastructure, users will have to accept dataset terms and conditions on the new system. Users' profiles including licensing information are not being transferred. In practical terms, this means for example that if you are downloading ERA5 data (web or API) from the current CDS, you will need to login to the new CDS and accept the terms and conditions for ERA5 (i.e. Licence to use Copernicus Products).
API Service - programmatic way of downloading data:
- A new version of the cdsapi package has been released. API users need to upgrade their CDS API (i.e pip install --upgrade cdsapi). Note that the new version of the cdsapi package works on both the current and new CDS.
- When EW- DS Beta is open, log in to the EW- DS Beta interface using your ECMWF account and follow the instructions online. Make sure you also accept the Terms and Conditions of the datasets you wish to download (see CDS/ADS/EW-DS Catalogue section above).
- You will need to update your .cdsapirc file as shown in the first black box displayed on the EW-DS Beta API page (you must be logged in!).
- Before running your API script to fetch data from EW-DS Beta, please use the "Show API request code" tool on the dataset Download Form on EW-DS Beta to check you are using the correct syntax for your API request. CDS/ADS/EW-DS API syntax is changed and some keys or parameter names may have also changed.
CDS Toolbox, Viewers and Applications:
The CDS Toolbox will be discontinued and is not migrated to the new CDS/EW-DS infrastructure. Support will continue to be provided to CDS Toolbox users until the CDS Toolbox is shut down in September 2024. ECMWF Support will however not be able to help with supporting users migrating their CDS Toolbox workflows onto another programming language/software.
As a consequence of the CDS Toolbox being discontinued, the Viewer tool for the following datasets in the CDS will also be discontinued.
New tool packages will be made available to provide software tools for weather and climate workflows that simplify data access, analysis, visualisation and much more. Please watch the user forum for announcements on this topic in the coming months.
All Climate Data Store Applications will also be retired and no longer available when the current Climate Data Store is shut down in September 2024.
CDS-Beta will have some better performing applications including the Copernicus Interactive Climate Atlas and Climate Pulse. More applications will be published including new versions of some applications in the legacy CDS e.g. ERA5 explorer, Global temperature trend monitor...
IP address range for CDS-beta
CDS Download servers are all still under the same IP range.
In the CDS-Beta, IP address ranges are:
136.156.136.0/21
136.156.128.0/21
Feedback and reporting bugs/issues:
Please send your feedback and/or report any issue/bug you may have encountered while using the new CDS/ADS/EW-DS infrastructure by raising an enquiry ticket through our dedicated Support Portal (ECMWF login required) - make sure you select "Yes" to the question Is this request related to CDS-beta/ADS-beta/CEMS-EW-beta? on the Support Portal request form - this will help the Support team with triaging enquiries more efficiently.
Good to know
- New publication of datasets (including publication of existing dataset new versions) will only be made in the new CDS/ADS/EW-DS infrastructure. Users are therefore urged to migrate over to the new CDS/ADS/EW-DS infrastructure as soon as possible.
- Some observation datasets will not be immediately available to download from the new CDS infrastructure while technical issues with these particular datasets are being addressed.
- Some unsupported datasets in current CDS will be deprecated in the new CDS infrastructure.
- For the time being, DOIs of datasets in CDS-Beta will point to the same datasets in the current CDS until the current CDS is shut down in September 2024.
- The current CDS and ADS will be completely shut down in September 2024. Please migrate to using the new CDS/ADS/EW-DS by September 3rd, 2024. Should users still have API scripts pointing to the retired CDS and ADS systems, these will systematically break and no support will be provided.
- For more information on specific changes to the CDS and ADS systems, visit: Informative page in readiness for CDS-Beta