Contributors: J. El Kassar (FUB), R. Preusker (FUB)
Issued by: Rene Preusker / Jan R. El Kassar
Date: 07/10/2019
Ref: C3S_312b_Lot1.2.3.4_v1.0_201908 _SQAD_TCWV_v1.0.1
Official reference number service contract: 2018/C3S_312b_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
General definitions
In the scope of the Copernicus Climate Change Service (C3S), a Thematic Climate Data Record (TCDR) has a fixed end point, whereas an Interim Climate Data Record (ICDR) is extended continuously. The TCWV_MERIS_SSMI_TCDR product is a TCDR which consists of two satellite-based estimates of the Total Columnar Water Vapor (TCWV) of the atmosphere. TCWV is a measure of the total water vapor content of the atmosphere over a certain area [kg/m2] and is also referred to as Precipitable Water Vapor (PWV) or Integrated Water Vapor (IWV). Hereafter and throughout C3S this variable will be always called TCWV.
The two estimates used in this CDR are derived by two different instruments over two types of surface. Over land, data from the Medium Resolution Imaging Spectrometer (MERIS) are used. Over water, the Special Sensor Microwave/Imager (SSM/I) is used. All documents related to the TCWV_MERIS_SSMI_TCDR product are mostly limited to processing of MERIS data by the Freie Universität Berlin (FUB). The processing of SSM/I data have been provided by the German Weather Service (DWD) and were not part of the processing at FUB and are thus not discussed in this or any of the other supplementary documents.
Scope of the document
This document is the System Quality Assurance Document (SQAD). It describes all elements of the processing chain for the TCWV_MERIS_SSMI_TCDR product. This product is a satellite based estimate of the water vapor content of the atmosphere. It consists of two products of which one has been fully processed at the Freie Universität Berlin (FUB) and is delivered to the Climate Data Store (CDS) by the Copernicus Climate Change Service (C3S). In this document, the focus lies on the processing of total column water vapor (TCWV) processed over land surfaces from measurements by the Medium Resolution Imaging Spectrometer (MERIS). Furthermore the interface to the CDS and internal procedures for managing and communicating system maintenance and for user support are described.
Executive summary
The TCDR of Total Column Water Vapour (TCWV) from the Freie Universität Berlin (FUB) is an in-house product delivered to the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S).
The MERIS TCWV retrieval is based on the Cloud Aerosol and Water Vapor algorithm (CAWA) [Preusker et al., 2015] which was developed at the FUB within the Advanced Clouds, Aerosols and Water Vapour products for Sentinel 3/OLCI (CAWA) ESA Scientific Exploitation of Operational Missions Element Program (SEOM). The retrieval is applied to the MERIS instrument onboard the polar orbiting ENVISAT.c
The SSMI TCWV is provided by DWD and is retrieved by SSM/I and SSMIS instruments onboard several polar orbiting DMSP satellites.
The MERIS TCWV and SSMI TCWV datasets are combined by the FUB team and comprises of 10 years (2002-2012) L3 monthly mean time series. The CDR provided here includes monthly means of TCWV on a regular global latitude-longitude grid, merged from MERIS (over land) and SSM/I (over ocean).
1. System Overview
1.1 Processing Hardware and Software
1.1.1 Data Processing
All processing from Level 1b to Level 3 data has been conducted by the Institute of Space Sciences (WEW) at the Freie Universität Berlin (FUB). Please note that Level in the context of Satellite Data Processing refers to the level of processing and not the quality of the data and is not related to the Levels of User Support described in Section 5 (User Support).
WEW works with Dell Server Racks with Scientific Linux installed on each machine. The retrieval algorithm is written in Python 3.7 [D2] and Fortran. The infrastructure surrounding the retrieval algorithm is written in Bash [D3] and Python 3.7.
1.1.2 Data Storage at the Processing Site
The WEW uses Synology DiskStation DS2415+ NAS-Servers for Data Storage.
1.2 FUB Processing Diagram
The MERIS TCWV product is generated in three steps: from Level 1b to Level 2 MERIS TCWV, from Level 2 MERIS TCWV to Level 3 daily composites and from these daily composites to Level 3 monthly composites (Figure 1).
Figure 1: The schematic of the MERIS TCWV processing at FUB.
First Level 1b data provided by ESA are filtered pixel-wise with corresponding Level 2 flags provided by ESA. Then, the filtered Level 1b radiances are fed together with auxiliary data from ECMWF’s ERA Interim reanalysis into the algorithm.
Since all Level 2 data are stored in the irregular grid of the satellite-projection of the instrument, the swath-wise MERIS TCWV data are regridded onto a 0.05° and 0.5° regular grid. These are merged into daily composites. Daily composites are merged to monthly means. Monthly means are created in two different resolutions: a low resolution (0.5° x 0.5°) and a high resolution (0.05° x 0.05°). For both Level 3 processing steps an average and standard deviation of all valid values in a grid box are calculated.
A detailed description of the retrieval algorithm and the Level 3 Data Processing can be found in the Algorithm Theoretical Basis Document (ATBD) for TCWV_MERIS_SSMI_TCDR [D4].
1.3 Back-up
All data, from Level 2 TCWV in swath-projection and ?irregular resolution up to the Level 3 daily composites and monthly means in high and low grid-resolution are backed up in-house at storage facilities of the WEW.
1.4 Transfer to the Climate Data Store (CDS)
1.4.1 Delivery of the data to C3S
DWD maintains a data server within the scope of the Earth System Grid Federation (ESGF), subject to the index node at the German Climate Computing Centre (DKRZ). It can be accessed at https://esgf.dwd.de. All data provided for the CDS by DWD will be uploaded to this server. It can be accessed via GridFTP, HTTP, or OPeNDAP protocols. Exact addresses of the provided files will be provided to the CDS upon submission. This allows automated access to the files, e.g. via scripts handling the delivery chain of an order in the CDS from the server to the user. The OPeNDAP facilities include access to subsets in the uploaded files, such as single variables only, as requested by CDS users. We aim for a 99% availability of the data server, with 95% of the data transactions happening at a rate of 10 Mbit/s or faster.
1.4.2 The ESGF node and OPeNDAP server at DWD
The data server that is used for providing data to the CDS is integrated into the ESGF network. It allows downloading of files or (via OPeNDAP) subsets of data from outside the DWD firewall. Preliminary analysis of the required disk space for data sets provided by DWD within C3S_312b_Lot1 shows that the quota on the data server dedicated to DWD's efforts within C3S_312b_Lot1 will not be exceeded.
2. Upgrade cycle implementation procedure
Since the MERIS dataset has a fixed end point in April 2012 there are no plans to implement an upgrade cycle.
3. Procedures for reprocessing CDR's
There are no reprocessing procedures planned for now (2019/08). The processing of TCWV over coastal waters and water surfaces not covered by SSM/I are currently being discussed.
4. System maintenance and system failures
In general, any issues with IT infrastructure that either delays delivery of data or impedes access to data via the CDS will be documented in the Quarterly Report.
4.1 Interfaces to external data
This section discusses treatment of delays at interfaces to external data sources.
4.1.1 Brokered CDRs and ICDRs
It is beyond the scope of our responsibility to safeguard the stability of services and data servers of external providers. If we become aware of issues at their end that will impede the timely delivery of brokered data, we will notify C3S.
4.2 Interface to the CDS
This section discusses maintenance and outages of the data server hosting the files for access through the CDS.
4.2.1 Planned maintenance
Planned outages of the data server on which the CDS accesses the files are kept to a minimum. However, software updates may become necessary in order to comply with the ESGF network, or hardware may need to be maintained or replaced. Such planned outages will be communicated to the CDS as early as possible. Respective notifications will be issued by DWD's IT department and delivered to the DWD-based members of the Service Management Team of C3S_312b_Lot1, who will ensure prompt passing on to the CDS. We do not expect to contact users of data sets submitted by us directly, but only through this communication to CDS.
We are open to requests by C3S to schedule dates on which no planned maintenance involving an outage should be carried out. Although a respective commitment by the IT department cannot be guaranteed, we always strive to respect such requests.
4.2.2 Unplanned outages
Unexpected system outages will be treated according to internal regulations and procedures at DWD's IT department. This involves notification of the DWD-based members of the Service Management Team, who will in turn notify C3S of the outage as quickly as possible. The IT department will resolve the issue according to their procedures and notify C3S, again via the Service Management Team.
5. User support
5.1 Enquiries
The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base (CKB) (CAMS and C3S Knowledge Base (CKB)) or the Copernicus Climate Data Store desk (https://cds.climate.copernicus.eu/contact-us). This is known as the level-0 process of User Support.
Figure 2: The schematic of Copernicus User Support (Copernicus User Support Handbook, V2.1)
Contact and User Support process on JIRA service desk
If the level-0 support fails to answer the user's query, they may then make a request, which is sent to the Copernicus User Support (CUS) Service team at ECMWF. These (level-1) requests will be handled within 8 hours.
For any scientific and specialist enquiries that cannot be answered by the CUS team at ECMWF or addressed by the Copernicus Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).
Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. This step is handled on the JIRA 1 Service Desk established at ECMWF for the current project. In case of specific scientific issues, the enquiries will be channeled to the ECV and data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In case of brokered datasets, the data specialist may choose to contact the responsible team at the provider level if they cannot resolve the issue on their own. In each yearly quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets raised by users on the JIRA system in the Quarterly Report.
- JIRA: Project management software tool created by Atlassian company ↩
References
n/a
