Contributors: H. Konrad (DWD), N. Rahpoe (DWD), O. Bobryshev (DWD)
Issued by: Deutscher Wetterdienst / H. Konrad
Date: 31/03/2021
Ref: C3S_D312b_Lot1.2.1.4-v1.0_202103_SQAD_ECVPrecipitation_COBRA_v1.0
Official reference number service contract: 2018/C3S_312b_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Acronyms
General definitions
n/a
Scope of the document
This document describes the systems used for delivering Copernicus micrOwave-based gloBal pRecipitAtion (COBRA) to the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S). Interfaces to external data, the infrastructure for data processing (if applicable), and the interface to the CDS are outlined. Information about new data cycles and reprocessing is also provided in this document. Internal procedures for managing and communicating system maintenance and for user support are described.
Executive summary
The Copernicus micrOwave-based gloBal pRecipitAtion (COBRA) Climate Data Record (CDR) is generated on a Linux-based multi-node batch system at DWD. The repositories for the various input data are listed. The final product is made accessible via DWD's webserver at https://c3s.satproj.klima.dwd.de/data/ and can be ordered via HTTP protocol. The data are password-protected. Login details are provided directly to the CDS team. Respective outages will be kept to a minimum and, whenever possible, communicated to C3S in advance.
We provide user support for queries relating to our products when forwarded to us by C3S (see Chapter 5 User Support).
1. System overview
1.1 System elements and interfaces
The Copernicus micrOwave-based gloBal pRecipitAtion (COBRA) product is a joint effort of CNR-ISAC and DWD within C3S. CNR-ISAC is responsible for the PNPR-CLIM algorithm development. The respective PNPR-CLIM output is eventually merged with HOAPS v4 observations produced within EUMETSAT's CMSAF. DWD is responsible for the production.
1.1.1 Multi-node batch system "C3S_312b"
DWD's IT department provides multi-node batch systems (MNBS) to working groups on request. These are capable of managing queues of c shell scripts. A dedicated MNBS has been implemented for the production of the present dataset.
The MNBS was used in the following processing steps during the production:
- Retrieval of precipitation rates by the PNPR-CLIM algorithm, applied to the FIDUCEO Fundamental Climate Data Record (FCDR) for AMSU-B/MHS
- Gridding of instantaneous PNPR-CLIM and HOAPS v4 observations onto an hourly 1° × 1° grid
- Merging and aggregating the hourly gridded values to daily and monthly data
These processing steps are computationally expensive because they involve handling large amounts of data. Therefore, it was necessary to parallelize the processing, for which the MNBS provided an optimal, easy-to-use solution.
1.1.2 Accompanying Linux systems
The MNBS (Section 1.1.1) is accessed through and reads and writes data on virtual Linux servers (CentOS version 7). The less expensive post-processing of the hourly gridded data (i.e., prior to merging and aggregating) could be handled directly on these virtual Linux servers.
1.1.3 Interfaces to external data
FIDUCEO FCDR for AMSU-B/MHS (PNPR-CLIM)
The PNPR-CLIM algorithm was applied to the FIDUCEO FCDR for AMSU-B and MHS, which was downloaded from CEDA:
https://catalogue.ceda.ac.uk/uuid/a8e9f44965434f3b861eba77688701ef
Auxiliary data (PNPR-CLIM)
Auxiliary data comprised temporally varying ERA5 output, downloaded from the CDS.
HOAPS v4 instantaneous observations
HOAPS v4 has been produced in the scope of CMSAF. Respective instantaneous observations were made available by the involved staff at DWD on the Linux system detailed in Section 1.1.2.
1.1.4 Delivery of the data to C3S
The data are made accessible via DWD's webserver at https://c3s.satproj.klima.dwd.de/data/. All data provided for the CDS by DWD will be uploaded to this server. It can be accessed via HTTP protocol. Exact addresses of the provided files will be provided to the CDS upon submission. This allows automated access to the files, e.g. via scripts handling the delivery chain of an order in the CDS from the server to the user. We aim for a 99% availability of the data server, with 95% of the data transactions happening at a rate of 10 Mbit/s or faster.
1.2 Hardware, supercomputers and cloud computing
1.2.1 Multi-node batch system "C3S_312b"
The MNBS consists of one managing server and 4 to 120 computing servers, each featuring 6 CPUs and 30 GB of memory. The hardware is an in-kind contribution by DWD's IT department and will return to the hardware pool when the processing in the scope of this product is finished. The number of available computing servers depends on general demand. Hence, no guarantees with respect to the number of computing nodes can be made, except for the minimum equipment of four. However, during our processing, we did not face shortages in computing power.
1.2.2 Accompanying Linux systems
The Linux system in which the MNBS is embedded, and in which certain post-processing steps and the communication with the webserver are carried out, is part of the DWD IT infrastructure. There are several virtual servers available. Their capacity is shared with other groups at DWD.
1.2.3 Back-up
There are three copies of the COBRA CDR dataset. The first copy is in the DWD Network Attached Storage. The second is the one accessible to the CDS on the http server, see Section 1.2.4. The third copy is stored at the ECFS (ECMWF's File Storage System), with a secure backup system. This ensures that we will be able to restore the data in the case of data corruption or loss of the data server.
1.2.4 Webserver at DWD
DWD hosts the webserver (CentOs Linux) that can be accessed from outside the DWD firewall via HTTP. Monitoring of server availability is routinely performed using Nagios IT Infrastructure Monitoring Tool. The check is performed every 60 second and if the data on the webserver are unavailable, an automatic email notification is issued.
Preliminary analysis of the required disk space for data sets provided by DWD within C3S_312b_Lot1 necessary disk space shows that the quota on the data server dedicated to DWD's efforts within C3S_312b_Lot1 will not be exceeded.
2. Upgrade cycle implementation procedure
Upgrades are currently not anticipated.
3. Procedures for reprocessing CDRs
Re-processing is currently not anticipated.
4. System maintenance and system failures
In the absence of an Interim Climate Data Record (ICDR) that would be delivered continuously, we do not expect a need for processing data after successful delivery to the CDS. Consequently, system maintenance and failure management are only relevant with respect to the interface to the CDS. In general, any issues with IT infrastructure that either delays delivery of data or impedes access to data via the CDS will be documented in the Quarterly Report.
4.1 Interface to the CDS
This section discusses maintenance and outages of the data server hosting the files for access through the CDS.
4.1.1 Planned maintenance
Planned outages of the data server on which the CDS accesses the files are kept to a minimum. However, software updates may become necessary, or hardware may need to be maintained or replaced. Such planned outages will be communicated to the CDS as early as possible. Respective notifications will be issued by DWD's IT department and delivered to the DWD-based members of the Service Management Team of C3S_312b_Lot1, who will ensure prompt passing on to the CDS. We do not expect to contact users of data sets submitted by us directly, but only through this communication to CDS.
We are open to requests by C3S to schedule dates on which no planned maintenance involving an outage should be carried out. Although a respective commitment by the IT department cannot be guaranteed, we always strive to respect such requests.
4.1.2 Unplanned outages
Unexpected system outages will be treated according to internal regulations and procedures at DWD's IT department. This involves notification of the DWD-based members of the Service Management Team, who will in turn notify C3S of the outage as quickly as possible. The IT department will resolve the issue according to their procedures and notify C3S, again via the Service Management Team.
5. User support
5.1 Enquiries
The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base (CKB) (CAMS and C3S Knowledge Base (CKB)) or the Copernicus Climate Data Store desk (https://cds.climate.copernicus.eu/contact-us). This is known as the level-0 process of User Support.
Figure 1: The schematic of Copernicus User Support (Copernicus User Support Handbook, V2.1)
Contact and User Support process on JIRA service desk
If the level-0 support fails to answer the user's query, they may then make a request, which is sent to the Copernicus User Support (CUS) Service team at ECMWF. These (level-1) requests will be handled within 8 hours.
For any scientific and specialist enquiries that cannot be answered by the CUS team at ECMWF or addressed by the Copernicus Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).
Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. This step is handled on the JIRA 1 Service Desk established at ECMWF for the current project. In case of specific scientific issues, the enquiries will be channeled to the ECV and data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In case of brokered datasets, the data specialist may choose to contact the responsible team at the provider level if they cannot resolve the issue on their own. In each yearly quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets raised by users on the JIRA system in the Quarterly Report.
- JIRA: Project management software tool created by Atlassian company ↩
References
n/a
