Contributors: J. Drücke (Deutscher Wetterdienst (DWD))
Issued by: J. Drücke (DWD)
Date: 27/09/2023
Ref: C3S2_D312a_Lot1.3.2.6_202309_SQAD_ECV_WaterVapour_COMBI_v1.2
Official reference number service contract: 2021/C3S2_312a_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
List of tables
List of figures
General definitions
Table 1: Definition of various terms used in the document
Variables | Abbreviation |
Brokered products | The C3S Climate Data Store (CDS) provides both data produced specifically for C3S and so-called brokered products. The later are existing products produced under an independent programme or project which are made available through the CDS. |
Climate Data Store (CDS) | The front-end and delivery mechanism for data made available through C3S. |
Scope of the document
This document describes the systems used for delivering the combined global near-infrared (NIR) and microwave imager (MW) Total Column Water Vapour data record (COMBI) brokered from EUMETSAT CM SAF to the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S). This is a brokered dataset and data processing is described in Algorithm Theoretical Basis Documents (ATBD) [D1-D3], the interface to the CDS is described in this document. Information about new data cycles and reprocessing are also provided in this document. Internal procedures for managing and communicating system maintenance, and for user support are described.
Executive summary
The COMBI Total Column Water Vapour product combines near-infrared (NIR)-based retrievals over land, coasts and sea-ice and CM SAF HOAPS data over open ocean. The NIR-algorithm development and the combination of NIR and HOAPS-based products has been (further) developed and carried out within the ESA CCI Water Vapour project (WV_cci).
The dataset is made accessible via DWD’s webserver at https://c3s.satproj.klima.dwd.de/data/ and can be ordered via HTTP protocol. The data are password-protected. Respective outages will be kept to a minimum and, whenever possible, communicated to C3S in advance.
We provide user support for queries relating to our products when forwarded to us by C3S.
1. System overview
1.1 System elements and interfaces
The interfaces for acquiring the satellite data and delivering the COMBI product TCDR to C3S are described in this section (see Figure 1-1 for schematic illustration).
Figure 1-1: Flowchart illustrating the process from data creation to its publication
1.1.1 Delivery of the data to C3S
The data are made accessible via DWD’s webserver at https://c3s.satproj.klima.dwd.de/. All data provided for the CDS by DWD will be uploaded as NetCDF files to this server as well as an associated list with addresses to relevant files. It can be accessed via HTTP protocol. Exact addresses of the provided files will be provided to the CDS team upon submission as well as the Login details. This allows automated access to the files, e.g. via scripts handling the delivery chain of an order in the CDS from the server to the user. We aim for a 99% availability of the data server, with 95% of the data transactions happening at a rate of 10 Mbit/s or faster.
1.2 Hardware, supercomputers and cloud computing
1.2.1 Computing infrastructure
The NIR-Level 2 processing and the combination of NIR and HOAPS-based products were carried out by Brockmann Consult within the ESA CCI Water Vapour project (WV_cci) while the HOAPS data was processed by CM SAF at DWD. Thus, the NIR and HOAPS L2 data, as well as the COMBI product, are produced outside the scope of C3S. Therefore, only data storage and delivery is described here. Please find additional information about the data production in the corresponding Algorithm Theoretical Basis Documents (ATBD): [D1] for HOAPS ATBD, [D2] for NIR product ATBD based MERIS, MODIS and OLCI instruments, and [D3] for TCWV COMBI product ATBD.
1.2.2 Backup
The COMBI product is not backed up within the framework of C3S. In the case of data corruption or loss on the data server, which serves as interface to the CDS, it can be downloaded again from its original, dedicated source, where back-ups outside our responsibilities are maintained.
1.2.3 Webserver at DWD
DWD hosts the webserver (CentOs Linux 7) that can be accessed from outside the DWD firewall via HTTP. Monitoring of server availability is routinely performed using Nagios IT Infrastructure Monitoring Tool. The check is performed every 60 seconds and if the data on the webserver are unavailable, an automatic email notification is issued.
Preliminary analysis of the required disk space for data sets provided by DWD within C3S2_312a_Lot1 necessary disk space shows that the quota on the data server dedicated to DWD’s efforts within C3S2_312a_Lot1 will not be exceeded.
2. Upgrade cycle implementation procedure
The CDR data products are brokered from external sources (EUMETSAT CM SAF) and released as a single delivery. Therefore, this section is not applicable. If there are future developments concerning this dataset which are not covered by C3S, it can be found at:
3. Procedures for reprocessing CDRs
The CDR data products are brokered from external sources (EUMETSAT CM SAF) and released as a single delivery. Therefore, this section is not applicable. If there are future developments concerning this dataset which are not covered by C3S, it can be found at:
4. System maintenance and system failures
In general, any issues with IT infrastructure that either delays delivery of data or impedes access to data via the CDS are documented in the Quarterly Implementation Reports.
4.1 Interfaces to external data
This section discusses treatment of delays at interfaces to external data sources.
4.1.1 Brokered data
It is beyond the scope of our responsibility to safeguard the stability of services and data servers of external providers. If we become aware of issues at their end that will impede the timely delivery of brokered data, we will notify C3S.
4.2 Interface to the CDS
This section discusses maintenance and outages of the data server hosting the files for access through the CDS.
4.2.1 Planned maintenance
Planned outages of the data server on which the CDS accesses the files are kept to a minimum. However, software updates may become necessary, or hardware may need to be maintained or replaced. Such planned outages will be communicated to the CDS as early as possible. Respective notifications will be issued by DWD’s IT department and delivered to the DWD-based members of the Service Management Team of C3S2_312a_Lot1, who will ensure prompt passing on to the CDS. We do not expect to contact users of data sets submitted by us directly, communication will be to the CDS only.
We are open to requests by C3S to schedule dates on which no planned maintenance involving an outage should be carried out. Although a respective commitment by the IT department cannot be guaranteed, we always strive to respect such requests.
4.2.2 Unplanned outages
Unexpected system outages will be treated according to internal regulations and procedures at DWD’s IT department. This involves notification of the DWD-based members of the Service Management Team, who will in turn notify C3S of the outage as quickly as possible. The IT department will resolve the issue according to their procedures and notify C3S, again via the Service Management Team.
5. User support
5.1 Enquiries
The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base (CKB) (CAMS and C3S Knowledge Base (CKB)) or the Copernicus Climate Data Store desk (https://cds.climate.copernicus.eu/contact-us). This is known as the level-0 process of User Support.
Figure 5-1: The schematic of Copernicus User Support (Copernicus User Support Handbook, V2.1)
5.2 Contact and User Support process on JIRA service desk
If the level-0 support fails to answer the user's query, they may then make a request, which is sent to the Copernicus User Support (CUS) Service team at ECMWF. These (level-1) requests will be handled within 8 hours.
For any scientific and specialist enquiries that cannot be answered by the CUS team at ECMWF or addressed by the Copernicus Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).
Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. This step is handled on the JIRA1 Service Desk established at ECMWF for the current project. In case of specific scientific issues, the enquiries will be channeled to the ECV and data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In case of brokered datasets, the data specialist may choose to contact the responsible team at the provider level if they cannot resolve the issue on their own. In each yearly quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets raised by users on the JIRA system in the Quarterly Report.
1 JIRA: Project management software tool created by Atlassian company
This document has been produced with funding by the European Union in the context of the Copernicus Climate Change Service (C3S), operated by the European Centre for Medium-Range Weather Forecasts on behalf on the European Union (Contribution Agreement signed on 22/07/2021). All information in this document is provided “as is” and no guarantee of warranty is given that the information is fit for any particular purpose. The users thereof use the information at their sole risk and liability. For the avoidance of all doubt, the European Commission and the European Centre for Medium-Range Weather Forecasts have no liability in respect of this document, which is merely representing the author’s view.
