Contributors: N. Clerbaux (Royal Meteorological Institute of Belgium (RMIB)), A. Velazquez Blazquez (RMIB), E. Baudrez (RMIB)
Issued by: RMIB/Nicolas Clerbaux
Date: 30/03/2023
Ref: C3S2_D312a_Lot1.3.2.5-v1.1_202303_SQAD_ECVEarthRadiationBudget_v1.3
Official reference number service contract: 2022/C3S2_312a_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
List of tables
List of figures
General definitions
Jargon | Definition |
Brokered products | The C3S Climate Data Store (CDS) provides both data produced specifically for C3S and so-called brokered products. The later are existing products produced under an independent programme or project which are made available through the CDS. |
Climate Data Store (CDS) | The front-end and delivery mechanism for data made available through C3S. |
OpenDAP | OpenDAP is an advanced software for remote data retrieval that simplifies all aspects of scientific data networking. This free software makes local data accessible to remote locations regardless of local storage format. |
Linux cron daemon | Cron is a time-based job scheduler in Unix-like operating systems. It allows users to schedule and automate the execution of recurring tasks or commands at specific intervals or times set by the user. |
THREDDS Data Server | The Thematic Real-time Environmental Distributed Data Services (THREDDS) is developed by Unidata. It aims at supporting students, educators and researchers with coherent access to a large collection of real-time and archived datasets from a variety of environmental data sources at a number of distributed server sites. The THREDDS Data Server is a web server that provides metadata and data access for scientific datasets, using a variety of remote data access protocols. Full description and software download via: |
Total Solar Irradiance (TSI) | The Total Solar Irradiance (TSI) quantifies the total amount of solar energy that is received by the Earth. It is defined per unit surface perpendicular to the Sun–Earth direction at the mean Sun–Earth distance. The TSI is a fundamental variable governing the climate system, and is recognized as Essential Climate Variable (ECV) by the Global Climate Observing System (GCOS). |
Scope of the document
This document describes the systems that are used to provide the following two Climate Data Records (CDRs) to the Copernicus Climate Change Service (C3S) Climate Data Store (CDS):
The version v01r02 of the Daily High-resolution Infrared Radiation Sounder (HIRS) Outgoing Longwave radiation (OLR) Climate Data Record (CDR) that is brokered from National Oceanic and Atmospheric Administration (NOAA) / National Center for Environmental Information (NCEI).
The version v3 of the Daily Total Solar Irradiance (TSI) CDR that is generated for the C3S in the frame of the C3S2_D312a_Lot1 contract.
The document also provides information on the procedures used in the event of problems or outages in the systems, whether planned or unplanned, and information on the provision of user support.
Executive summary
The System Quality Assurance Document (SQAD) details the technical aspects related to the provision of two different Earth Radiation Budget (ERB) data records to the CDS.
The first CDR is the version v01r02 of the Daily High-resolution Infrared Radiation Sounder (HIRS) Outgoing Longwave radiation (OLR) Climate Data Record (CDR) that is brokered from National Oceanic and Atmospheric Administration (NOAA) / National Center for Environmental Information (NCEI). Given the brokered nature of this record, the SQAD only details how the data is brokered and not how the data is generated at NOAA/NCEI.
The second CDR is the version 3 of the daily TSI composite which is generated for the C3S in the frame of the C3S2_D312a_Lot1 contract. The SQAD fully describes the system implemented to produce this CDR and its corresponding Interim Climate Data Record (ICDR).
Back-end user support is provided by a point of contact at RMIB (see details about user support in Section 6), ensuring a response to product related user queries within the allocated time frame and meeting the relevant Key Performance Indicator (KPI) targets.
1. System overview
1.1 Overview of the HIRS Daily OLR CDR
The NOAA/NCEI, the producer of the HIRS OLR data, provides the following executive summaries for this CDR1: The daily Outgoing Longwave Radiation (OLR) Climate Data Record (CDR) measures the amount of terrestrial radiation released into space and, by extension, the amount of cloud cover and water vapor that intercepts that radiation in the atmosphere. Input data for the daily OLR record primarily comes from the high-resolution infrared radiation sounder (HIRS). The final record is generated through a combination of statistical techniques, including OLR regression, instrument ambient temperature prediction coefficients, and inter-satellite bias corrections.
This Climate Data Record (CDR) contains the daily mean Outgoing Longwave Radiation (OLR) time series in global 1 degree x 1 degree equal-angle gridded maps spanning from January 1, 1979 to December 31, 2013, and continuing daily with a two-day lag. The OLR is estimated directly from the HIRS radiance observations for all sky conditions. The observations from imagers onboard international operational geostationary satellites are incorporated to improve the sampling of the OLR diurnal variation. The Daily OLR CDR is at its initial version 1.2. The data file format is netCDF-4 with CF metadata, and it is accompanied by algorithm documentation, data flow diagram and source code for the NOAA CDR Program.
This version (v01r02 or simply “1.2”) of the OLR-daily CDR contains more than 40 years of satellite-based measurements of OLR, on a global scale with a 1° x 1° spatial resolution. The HIRS Daily OLR CDR v01r02 is made available to the C3S Climate Data Store through an OPeNDAP server hosted by the NOAA at:
https://www.ncei.noaa.gov/thredds/catalog/cdr/olr-daily/catalog.html
As the OpenDAP functionality is not (yet) supported by the CDS, the daily data are accessed through a “go-between” server hosted at the Royal Meteorological Institute of Belgium (RMIB). See details in Section 1.4.
1.2 System elements and interfaces
The HIRS OLR CDR is generated and archived outside of this contract by the NOAA/NCEI. Hence, this section only describes the interfaces for delivering the data to the C3S CDS. The C3S CDS can access the gridded daily mean OLR via the NOAA/NCEI THREDDS server at:
https://www.ncei.noaa.gov/thredds/catalog/cdr/olr-daily/catalog.html
This data server allows users to directly browse and subset data products, and provides additional functionalities such as the geographical subsetting and accessing the file contents directly via the internet from data processing scripts. The latter functionality is intended for use by the C3S CDS and similar services working in pull mode.
Several data providers/brokers have suggested that the CDS should use the OpenDAP protocol when accessing data on remote servers in pull mode, instead of hosting local copies of the CDRs. This is in particular requested to maintain statistics of the data usage as each C3S user’s requests would be “seen” by the OpenDAP server. However, this functionality is not yet implemented by the CDS, the daily data are accessed through a “go-between” server hosted at the Royal Meteorological Institute of Belgium (RMIB). The data flow is illustrated in Figure 1-1 (the HIRS OLR CDR is located on the NOAA OpenDAP server).
Figure 1-1: data flowchart.
In the case of unforeseen system failures, or other problems affecting data delivery through the OPeNDAP server, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS.
1.3 Hardware and data server system – NOAA / NCEI server
The distribution of the NOAA/NCEI CDRs is performed via the NOAA/NCEI OPeNDAP server which runs the THREDDS Data Server [Version 4.6.11 - 2017-12-04T16:22:46-0700] software providing OPeNDAP server functionality. This server is accessed via:
https://www.ncei.noaa.gov/thredds/
The HIRS daily OLR data are organized in yearly files, e.g.,
https://www.ncei.noaa.gov/thredds/dodsC/cdr/olr-daily/olr-daily_v01r02_19790101_19791231.nc.html
(…)
https://www.ncei.noaa.gov/thredds/dodsC/cdr/olr-daily/olr-daily_v01r02_20210101_20211231.nc.html
The last file(s), with “preliminary” in their names, are an interim version that correspond to the Interim Climate data record (ICDR) concept.
1.4 Hardware and data server system – RMIB “go-between” server
The NOAA/NCEI server provides the daily OLR data organized in yearly files. To be used by the CDS’s toolkits, it is however requested that each data file contains a single time step. Therefore, to be accessed by the CDS, the data has to be separated in daily files. This is performed on the RMIB “go-between” server, as shown on the flowchart in Figure 1-1.
These separated daily HIRS OLR files can be accessed on the RMIB “go-between” server. The daily data files can be access via the URL:
https://gerb.oma.be/c3s/data/hirs-olr-daily/tcdr/v1.2/olr/YYYY/MM/DD
where YYYY, MM, and DD are the calendar year, month and day. For instance:
https://gerb.oma.be/c3s/data/hirs-olr-daily/tcdr/v1.2/olr/1987/06/12
The preliminary data files (i.e. ICDR) are available in a separate branch (icdr) of the directories’ tree:
https://gerb.oma.be/c3s/data/hirs-olr-daily/icdr/v1.2/olr/YYYY/MM/DD
e.g., https://gerb.oma.be/c3s/data/hirs-olr-daily/icdr/v1.2/olr/2023/01/20
The creation of the daily files on the RMIB “go-between” server is done using the NetCDF Subset Service (NCSS) of NOAA/NCEI available at (for the year 1979 in this example):
Specifically, the command to download a single day takes the form shown in the box below (where YYYY,MM,DD are the year, month and day). This command is executed for all the day/month/year which are present on the yearly files on the NOAA server.
A script is run automatically each night (at 05:00 UTC) and new/change to the yearly files on the NOAA server are detected. In case of new/updated file, the daily data corresponding to this year are extracted using the command in the box below. In practice, this daily extraction concerns the preliminary data from which new data are available (approximately) each day and are added to the icdr part of the go-between server. Once per year, the preliminary data are replaced by final data, and in this case also the corresponding daily data are extracted and added to the tcdr part of the go-between server.
Command used to extract the daily HIRS OLR files #!/bin/bash YYYY=2008 MM=11 DD=07 OUT=data_312a_Lot1_hirs-olr-daily_tcdr_v1.2_olr_YYYY_MM_DD.nc URL=https://www.ncei.noaa.gov/thredds/ncss/cdr/olr-daily FILE=olr-daily_v01r02_${YYYY}0101_${YYYY}1231.nc VAR=”var=olr” T_START=”time_start=${YYYY}-${MM}-${DD}T00%3A00%3A00Z” T_END=”time_end=${YYYY}-${MM}-${DD}T23%3A59%3A59Z” wget –O ${OUT} ‘${URL}/${FILE}?${VAR},${T_START}?${T_END}’ |
|---|
Following this request and the retrieval of the daily data file onto the RMIB server, the following 3 global attributes within the NetCDF file have to be changed as the output file contains a single day of data and not the full year anymore. This is done using ‘cdo’ for the following variables which are changed from:
:time_coverage_start = "1979-01-01T00:00:00.00Z" ;
:time_coverage_end = "1979-12-31T23:59:59.99Z" ;
:time_coverage_duration = "P365D" ;
to:
:time_coverage_start = "1979-01-01T00:00:00.00Z" ;
:time_coverage_end = "1979-01-01T23:59:59.99Z" ;
:time_coverage_duration = "P1D" ;
The cdo commands are executed just after the data download, thus just after the ‘wget’ command in the script above.
2. System overview – Generation of daily TSI CDR (v3)
2.1 Overview of the C3S daily TSI CDR v3.x
The C3S daily Total Solar Irradiance (TSI) timeseries is made available to the C3S Climate Data Store (CDS) through an HTTPS server hosted by the Royal Meteorological Institute of Belgium (RMIB). The server provides a “fixed” validated version of the TCDR over the period 1st Jan. 1979 – 31st Dec. 2020 (in version 3.0) and regularly updates the ICDR with an increasing version number v3.x. Please note, that the ICDR addresses listed here, may not be active yet, but will be available soon after respective ICDR deliveries.
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_TCDR_v3.0.txt
https://gerb.oma.be/tsi/C3S_RMIB_daily _TSI_composite_ICDR_v3.1.txt
https://gerb.oma.be/tsi/C3S_RMIB_daily _TSI_composite_ICDR_v3.2.txt
(…)
Furthermore, the “latest” ICDR provided automatically incorporates new data on a daily basis:
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_ICDR_v3_latest.txt
The main processing consists in the generation of the TSI daily composite timeseries from the individual instruments timeseries as described in the ATBD [D1].
For the ICDR, a daily download of new TSI data is performed for the “active” instruments before launching the TSI composite processing. Currently the “active” instruments are: DIARAD/VIRGO, PMO06/VIRGO, and TIM/TSIS. This list is likely to change in the future due to the old age of DIARAD and PMO06 and also due to the incorporation of new TSI measurements.
2.2 System elements and interfaces
For the still operational TSI instruments, daily downloads of fresh data are performed using the wget command. These downloads are launched automatically by the Linux cron daemon at 05:00 UTC. Then, the ICDR is updated to incorporate the new data.
The C3S TSI TCDR/ICDR are generated and archived at the Royal Meteorological Institute of Belgium (RMIB). The C3S CDS can access the daily TSI timeseries, using simple HTTP or FTP requests, at:
2.3 Hardware and data server system
2.3.1 Computing infrastructure
The data downloads (via wget) and the TCDR and ICDR processing are performed on the RMIB computing server “typhoon”. A hot backup of the processing is performed on the backup computing server “tornado”. The “typhoon” and “tornado” are physical Linux servers located in the server room of the RMIB “Computer Center”.
2.3.2 Data server system – Input data
For the still operational instruments, the input data are downloaded on a daily basis from the following ftp/http servers:
DIARAD/VIRGO: Access on the “solar.oma.be” server.
PMO6/VIRGO2: ftp://ftp.pmodwrc.ch/pub/data/irradiance/virgo/
TIM/TSIS: http://lasp.colorado.edu/data/tsis/tsi_data/tsis_tsi_L3_c24h_latest.txt
Data sources for the instruments that are not operational anymore are specified in the ATBD [D1] under the “Input Data” Section.
2.3.3 Data server system – Generated composite data
The results are then copied onto the RMIB GERB HTTPS server (https://gerb.oma.be) for data dissemination. The CDS accesses the TSI data on this server. The “gerb.oma.be” HTTPS server is running on a virtual machine (VM).
3. Upgrade cycle implementation procedure
3.1 HIRS OLR Brokering
The CDR data products are brokered from external sources (the NOAA/NCEI team) and released as a single delivery. Therefore, no formal upgrade cycle implementation procedure is defined.
Close contact is maintained with the HIRS OLR science team. Future upgrades will likely take the form of a change in data version (currently v1.2). Although a release date is not yet available, a reprocessing of the HIRS OLR is foreseen to incorporate HIRS-like OLR from the CrIS and IASI instruments. From the CDS brokering point of view, this will likely just necessitate a version increase, and an upgrade of the PQAD.
An upgrade may be done should the OpenDAP mechanism become supported by the CDS. However, no formal procedure is “pro-actively” defined for this. Such an upgrade will likely be implemented during a change of contract phase (currently C3S2).
3.2 Daily TSI v3 CDR
The ICDR is updated automatically each day to incorporate new data, at 05:00 UTC. The resulting TSI timeseries is labeled as ‘latest’:
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_ICDR_v3_latest.txt
There is no versioning in the name of this data file. This stresses the fact that it is generated automatically, without any quality check. For traceability, the date of the processing is included in the header, at the beginning of the data file.
Regularly (each 6 months), the ICDR is validated and then released as a new version of the ICDR. Please note, that the ICDR addresses listed here, may not be active yet, but will be available soon after respective ICDR deliveries.
https://gerb.oma.be/tsi/C3S_RMIB_daily _TSI_composite_ICDR_v3.1.txt
https://gerb.oma.be/tsi/C3S_RMIB_daily _TSI_composite_ICDR_v3.2.txt
(…)
Validation results (KPI compliance) are included in the Quarterly Report (QR).
4. Procedures for reprocessing CDR’s
4.1 HIRS OLR Brokering
As there is no HIRS instrument on the latest satellites (MetOp-C, NOAA-20 and NOAA-21), a reprocessing of the HIRS OLR is foreseen to incorporate HIRS-like OLR from the CrIS and IASI instruments. The HIRS OLR team at University of Maryland is working on this reprocessed CDR, but a date for the formal release is not yet announced.
From the CDS brokering point of view, this will just necessitate an increase of the version (currently v1.2), and an upgrade of the PQAD.
4.2 HIRS OLR Brokering
This is already the third version of the Daily TSI TCDR timeseries produced for C3S. The older versions remain available at:
https://gerb.oma.be/tsi/RMIB_TSI_composite_TCDR_v1.0.txt
https://gerb.oma.be/tsi/RMIB_TSI_composite_TCDR_v2.0.txt
The following table 4-1 summarizes the main changes between the versions.
Table 4-1: Changes between previous version
Version | Changes with respect to previous version |
1.0 |
|
2.0 |
|
3.0 |
|
5. System maintenance and system failures
5.1 HIRS OLR Brokering
The data products brokered from the NOAA/NCEI are served to the C3S CDS (pull mode) from an OPeNDAP server hosted by NOAA/NCEI: https://www.ncei.noaa.gov/thredds. However, as long as OPeNDAP functionality is not implemented in the CDS, the RMIB’s go-between server is used:
https://gerb.oma.be/c3s/data/hirs-olr-daily/tcdr/v1.2/olr/YYYY/MM/DD
where YYYY, MM, DD are the requested year, month and day respectively. The planned outages of the server, and therefore the availability of data, will be communicated as far as possible to the CDS well in advance. Any notification procedures requested by the CDS will be complied with.
The server is monitored by the RMIB in the framework of the C3S2_312a_Lot1 contract. In the case of unforeseen system failures, or other problems affecting data delivery through the OPeNDAP server, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS. For longer outages (longer than a few hours) the CDS will be notified again once the issue has been resolved.
Issues or outages, whether planned or unplanned, will be documented in the relevant Quarterly Report.
5.2 Daily TSI v3.x CDR
The planned outages of the server, and therefore the availability of data, will be communicated as far as possible to the CDS well in advance. Any notification procedures requested by the CDS will be complied with.
The servers are monitored by the RMIB in the frame of the C3S2 312A Lot1 contract. In the case of unforeseen system failures, or other problems affecting data delivery through the HTTPS servers, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS. For longer outages (longer than a few hours) the CDS will be notified again once the issue has been resolved.
Issues or outages, whether planned or unplanned, will be documented in the Quarterly Report.
Note that the TSI composite processing is running in parallel on 2 physical servers: “typhoon” (operational server) and “tornado” (backup server). The likelihood of simultaneous problem is therefore relatively low, although several single-point-of-failures are identified (e.g. network switch, FTP/HTTP virtual machine).
The access to the input data (individual timeseries) is verified before execution of the composite processing. In case of any data access problem to one of these input files, the “latest” ICDR is not updated and error messages are sent (via email) to the relevant RMIB scientists for action.
6. User support
The point of contact will ensure that each product related user query is redirected to the most appropriate science team member for the formulation of a response. The response will then be delivered to the user within the appropriate time frame. Throughout the process, the point of contact will ensure that the relevant key performance indicator (KPI) targets are met, and that the information collected from the science team will be used to develop the user knowledge base (via user documentation, FAQs, and user guides).
6.1 Enquiries
The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base on Copernicus Climate Data Store desk. This is known as the level-0 process of User Support.
Figure 6-1: The schematic of Copernicus User Support (CUS Handbook V2).
6.2 Contact and User Support process on JIRA Service Desk
If the level-0 support fails to answer the user´s query, they may then make a request which is sent to the Copernicus User Support (CUS) Service team at ECMWF. These (level-1) requests will be handled within 8 hours.
For any scientific and specialist enquiries that cannot be answered by the CUS team at ECMWF or addressed by the Copernicus Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2)
Enquiries forwarded to the Copernicus User Support Specialist team at DWD will be acknowledged within 3 working days (target 100%) and a notification sent to the user. This step is handled on the JIRA3 Service Desk established at ECMWF for the current project. In case of specific scientific issues, the enquiries will be channelled to the data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In case of brokered datasets, the data specialist may choose to contact the responsible team at the provider level if they cannot resolve the issue on their own. In each yearly quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with “1” (very unsatisfied) to “5” (very satisfied). We will also list the number of tickets raised by users on the Jira system in the Quarterly Report.
