Contributors: AN. Velazquez Blazquez Clerbaux (Royal Meteorological Institute of Belgium (RMIB)), NA. Clerbaux Velazquez Blazquez (RMIB), E. Baudrez (RMIB), C. Aebi (RMIB)

Issued by: RMIB/Clerbaux

Date: 0823/0702/20212023

Ref: C3SC3S2_D312bD312a_Lot1.3.2.5.41-v1.0_201903202205_SQAD_ECVEarthRadiationBudget_v1.1

Official reference number service contract: 20182021/C3SC3S2_312b312a_Lot1_DWD/SC1

Easy Heading Macro
navigationExpandOption collapse-all-but-headings-1-2

History of modifications

List of datasets covered by this document

Related documents

Related documents

Acronyms

General definitions

Scope of the document

This document describes the systems that are used for brokering the HIRS OLR CDR data from NOAA/NCEI to the Copernicus Climate Change Service (C3S) Climate Data Store (CDS). It also provides information on the procedures used in the event of problems or outages in the systems, whether planned or unplanned, and information on the provision of user support.

Executive summary

...

List of tables

List of figures

General definitions

Scope of the document

In this first version, this document describes the systems that are used for brokering the High-Resolution Infrared Radiation Sounder (HIRS) Outgoing Longwave radiation (OLR) Climate Data Record (CDR) data from National Oceanic and Atmospheric Administration (NOAA) / National Center for Environmental Information (NCEI) to the Copernicus Climate Change Service (C3S) Climate Data Store (CDS). It also provides information on the procedures used in the event of problems or outages in the systems, whether planned or unplanned, and information on the provision of user support.

Executive summary

The NOAA/NCEI, the producer of the HIRS OLR data, provides the following executive summaries for this CDR²: The daily Outgoing Longwave Radiation (OLR) Climate Data Record (CDR) measures the amount of terrestrial radiation released into space and, by extension, the amount of cloud cover and water vapor that intercepts that radiation in the atmosphere. Input data for the daily OLR record primarily comes from the high-resolution infrared radiation sounder (HIRS). The final record is generated through a combination of statistical techniques, including OLR regression, instrument ambient temperature prediction coefficients, and inter-satellite bias corrections.

This Climate Data Record (CDR) contains the daily mean Outgoing Longwave Radiation (OLR) time series in global 1 degree x 1 degree equal-angle gridded maps spanning from January 1, 1979 to December 31, 2013, and continuing daily with a two-day lag. The OLR is estimated directly from the HIRS radiance observations for all sky conditions. The observations from imagers onboard international operational geostationary satellites are incorporated to improve the sampling of the OLR diurnal variation. The Daily OLR CDR is at its initial version 1.2. The data file format is netCDF-4 with CF metadata, and it is accompanied by algorithm documentation, data flow diagram and source code for the NOAA CDR Program.

This version (v01r02 or simply "1.2") of the OLR-daily CDR contains more than 40 years of satellite-based measurements of OLR, on a global scale with a 1° x 1° spatial resolution. The HIRS Daily OLR CDR v01r02 is made available to the C3S Climate Data Store through an OPeNDAP server hosted by the NOAA at:

https://www.ncei.noaa.gov/thredds/catalog/cdr/olr-daily/catalog.html

As the OpenDAP functionality is not (yet) supported by the CDS, the daily data are accessed through a "go-between" server hosted at the Royal Meteorological Institute of Belgium (RMIB). This is described in section 1.3 of this document.
In the case of unforeseen system failures, or other problems affecting data delivery through the OPeNDAP server, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS.

Back-end user support is provided by a point of contact at RMIB (see details about user support in Section 5), ensuring a response to product related user queries within the allocated time frame and meeting the relevant Key Performance Indicator (KPI) targets.

1. System overview

1.1 System elements and interfaces

The HIRS OLR CDR is generated and archived outside of this contract by the NOAA/NCEI. Hence, this section only describes the interfaces for delivering the data to the C3S CDS. The C3S CDS can access the gridded monthly mean OLR via the NOAA/NCEI OPeNDAP server at:
https://www.ncei.noaa.gov/thredds/dodsC/cdr/olr-monthly/olr-monthly_v02r07_197901_202002.nc.html
This data server allows users to directly browse and subset data products, and provides additional functionalities such as accessing file contents directly via the internet from data processing scripts. The latter functionality is intended for use by the C3S CDS and similar services working in pull mode.

1.2 Hardware and data server system

As said, the HIRS OLR CDR is generated and distributed outside of this contract. For this reason, the computer infrastructure for generating and storing the data is not described in this SQAD.

The distribution to the NOAA/NCEI CDRs is performed via the NOAA/NCEI OPeNDAP server which runs the THREDDS Data Server [Version 4.6.11 - 2017-12-04T16:22:46-0700] software providing OPeNDAP server functionality. This server is accessed via:

...

...

1. System overview

1.1 System elements and interfaces

The HIRS OLR CDR is generated and archived outside of this contract by the NOAA/NCEI. Hence, this section only describes the interfaces for delivering the data to the C3S CDS. The C3S CDS can access the gridded daily mean OLR via the NOAA/NCEI THREDDS server at:

https://www.ncei.noaa.gov/thredds/catalog/cdr/olr-daily/catalog.html

This data server allows users to directly browse and subset data products, and provides additional functionalities such as the geographical subsetting and accessing of the file contents directly via the internet from data processing scripts. The latter functionality is intended for use by the C3S CDS and similar services working in pull mode.

Several data providers/brokers have suggested that the CDS should use the OpenDAP protocol when accessing data on remote servers in pull mode, instead of hosting local copies of the CDRs. This is in particular requested to maintain statistics of the data usage as each C3S user request would be "seen" by the OpenDAP server. However, as this functionality is not yet implemented by the CDS, the daily data are accessed through a "go-between" server hosted at the Royal Meteorological Institute of Belgium (RMIB). The data flow is illustrated in Figure 1 (the HIRS OLR CDR is located on the NOAA OpenDAP server).

Image Added
Figure 1: Data flowchart illustrating how the RMIB "go-between" server pulls data from the NASA and NOAA OpenDAP compliant servers.

1.2 Hardware and data server system – NOAA / NCEI server

The distribution of the NOAA/NCEI CDRs is performed via the NOAA/NCEI OPeNDAP server which runs the THREDDS Data Server [Version 4.6.11 - 2017-12-04T16:22:46-0700] software providing OPeNDAP server functionality. This server is accessed via:

https://www.ncei.noaa.gov/thredds/

The HIRS daily OLR data are organized in yearly files. These currently run from 1979 to 2021, so the 1^st and last files in the series at the moment are:

  https://www.ncei.noaa.gov/thredds/dodsC/cdr/olr-daily/olr-daily_v01r02_19790101_19791231.nc.html

(…)

  https://www.ncei.noaa.gov/thredds/dodsC/cdr/olr-daily/olr-daily_v01r02_20210101_20211231.nc.html

In addition to these annual files, an interim version (with “preliminary” in the name) that corresponds to the Interim Climate data record (ICDR) concept are also made available, e.g.

https://www.ncei.noaa.gov/thredds/dodsC/cdr/olr-daily/olr-daily_v01r02-preliminary_20220101_20220521.nc.html

1.3 Hardware and data server system – RMIB "go-between" server

The NOAA/NCEI server provides the daily OLR data organized in yearly files. To be used by the CDS’s toolkits, it is however requested that each data file contains a single time step. Therefore, to be accessed by the CDS, the data has to be separated into daily files. This is performed on the RMIB “go-between” server, as shown on the flowchart in Figure 1.

These separated daily HIRS OLR files can be accessed on the RMIB “go-between” server. The daily data files can be access via the generic URL which is of the form:

“https://gerb.oma.be/c3s/data/hirs-olr-daily/tcdr/v1.2/olr/YYYY/MM/DD”

where YYYY, MM, and DD are the calendar year, month and day. For instance:

https://gerb.oma.be/c3s/data/hirs-olr-daily/tcdr/v1.2/olr/1987/06/12

returns data for the 12^th of June 1987.

The preliminary data files (i.e. ICDR) are available in a separate branch (icdr) of the directories’ tree:

" https://gerb.oma.be/c3s/data/hirs-olr-daily/icdr/v1.2/olr/”

with “preliminary” added to the file name:

“https://gerb.oma.be/c3s/data/hirs-olr-daily/icdr/v1.2/olr/YYYY/MM/DD”

 e.g., https://gerb.oma.be/c3s/data/hirs-olr-daily/icdr/v1.2/olr/2023/01/20

returns data for the 20^th of January 2023.

The creation of the daily files for upload on the RMIB “go-between” server is done using the NetCDF Subset Service (NCSS) of NOAA/NCEI available at (for the year 1979 in this example):

https://www.ncei.noaa.gov/thredds/ncss/grid/cdr/olr-daily/olr-daily_v01r02_19790101_19791231.nc/dataset.html

Specifically, the command to download a single day takes the form shown in the box below (where YYYY,MM,DD are the year, month and day). This command is executed for all the day/month/year which are present in the yearly files on the NOAA server.

A script is run automatically each night (at 05:00 UTC) and any new files or changes to existing yearly files on the NOAA server are detected. In case of a new/updated file, the daily data corresponding to the relevant year are extracted using the command in the box below. In practice, this daily extraction concerns the preliminary files from which new data are available (approximately) each day and are added to the icdr part of the go-between server. Once per year, the preliminary data are replaced by final data, and in this case also the corresponding daily data are extracted and added to the tcdr part of the go-between server.

Command used to extract the daily HIRS OLR files #!/bin/bash YYYY=2008 MM=11 DD=07 OUT=data_312a_Lot1_hirs-olr-daily_tcdr_v1.2_olr_YYYY_MM_DD.nc URL= https://www.ncei.noaa.gov/thredds/ncss/cdr/olr-dailyImage Added FILE=olr-daily_v01r02_${YYYY}0101_${YYYY}1231.nc VAR="var=olr" T_START="time_start=${YYYY}${MM}${DD}T00%3A00%3A00Z" T_END="time_end=${YYYY}${MM}${DD}T23%3A59%3A59Z" wget –O ${OUT} '${URL}/${FILE}?${VAR},${T_START}?${T_END}'

Following this request and the retrieval of the daily data file onto the RMIB server, the following 3 global attributes within the NetCDF file have to be changed as the output file contains a single day of data and not the full year anymore. This is done using 'cdo' for the following variables which are changed from

:time_coverage_start = "1979-01-01T00:00:00.00Z" ;
:time_coverage_end = "1979-12-31T23:59:59.99Z" ;
:time_coverage_duration = "P365D" ;

into

:time_coverage_start = "1979-01-01T00:00:00.00Z" ;
:time_coverage_end = "1979-01-01T23:59:59.99Z" ;
:time_coverage_duration = "P1D" ;

The 'cdo' commands are executed just after the data download, thus just after the 'wget' command in the script above.

2. Upgrade cycle implementation procedure

The CDR data products are brokered from external sources (the NOAA/NCEI team) and released as a single delivery. Therefore, no formal upgrade cycle implementation procedure is defined..
Close contact is maintained with the HIRS OLR science team. Future upgrades will likely take the form of a change in data version (currently v1.2). Although a release date is not already available, a reprocessing of the HIRS OLR is foreseen to incorporate HIRS-like OLR from the CrIS and IASI instruments. From the CDS brokering point of view, this will likely just necessitate a version increase, and an upgrade of the PQAD.
An upgrade may be implemented should the OPeNDAP mechanism be supported by the CDS. However, no formal procedure is "pro-actively" defined for this. Such an upgrade will likely be implemented during a change of contract phase (currently C3S2).

3. Procedures for reprocessing CDR's

As there is no HIRS instrument on the latest relevant satellites (MetOp-C, NOAA-20 and NOAA-21), a reprocessing of the HIRS OLR is foreseen to incorporate HIRS-like OLR from the CrIS and IASI instruments. The HIRS OLR team at the University of Maryland is working on this reprocessed CDR, but a date for the formal release is not yet announced.
From the CDS brokering point of view, this will just necessitate an update of the version (currently v1.2), and an upgrade of the PQAD.

4. System maintenance and system failures

The data products brokered from the NOAA/NCEI are served to the C3S CDS (pull mode) from an OPeNDAP server hosted by NOAA/NCEI: https://www.ncei.noaa.gov/thredds. However, as long as OPeNDAP functionality is not implemented in the CDS, the RMIB's go-between server is used:

"https://gerb.oma.be/c3s/data/hirs-olr-daily/tcdr/v1.2/olr/YYYY/MM/DD"

where YYYY, MM, DD are the requested year, month and day respectively. The planned outages of the server, and therefore the availability of data, will be communicated as far as possible to the CDS well in advance. Any notification procedures requested by the CDS will be complied with.

The server is monitored by the RMIB in the framework of the C3S2_312a_Lot1 contract. In the case of unforeseen system failures, or other problems affecting data delivery through the OPeNDAP server, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS. For longer outages (longer than a few hours) the CDS will be notified again once the issue has been resolved.

Issues or outages, whether planned or unplanned, will be documented in the relevant Quarterly Report.

5. User support

The point of contact will ensure that each product related user query is redirected to the most appropriate science team member for the formulation of a response. The response will then be delivered to the user within the appropriate time frame. Throughout the process, the point of contact will ensure that the relevant key performance indicator (KPI) targets are met, and that the information collected from the science team will be used to develop the user knowledge base (via user documentation, FAQs, and user guides).

...

1.3 Documentation

2. Upgrade cycle implementation procedure

The CDR data products are brokered from external sources (the NOAA/NCEI team) and released as a single delivery. Therefore, this section is not applicable.

3. Procedures for reprocessing CDR's

The CDR data products are brokered from external sources (the NOAA/NCEI team) and released as a single delivery. Therefore, this section is not applicable.

4. System maintenance and system failures

The data products brokered from the NOAA/NCEI are served to the C3S CDS (pull mode) from an OPeNDAP server hosted by NOAA/NCEI: https://www.ncei.noaa.gov/thredds

The planned outages of the server, and therefore the availability of data, will be communicated as far as possible to the CDS well in advance. Any notification procedures requested by the CDS will be complied with.

The server is monitored by the RMIB in the frame of the C3S 312B Lot1 contract. In the case of unforeseen system failures, or other problems affecting data delivery through the OPeNDAP server, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS. For longer outages (longer than a few hours) the CDS will be notified again once the issue has been resolved.
Issues or outages, whether planned or unplanned, will be documented in the Quarterly Report.

...

5.1 Enquiries

The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base (CKB) (CAMS and C3S Knowledge Base (CKB)) or the Copernicus Climate Data Store desk (https://cds.climate.copernicus.eu/contact-us)Knowledge Base on Copernicus Climate Data Store desk. This is known as the level-0 process of User Support.

Image Modified

Figure 12:  The schematic of Copernicus User Support (Copernicus User Support CUS Handbook , V2 [D2]).1)

5.2 Contact and User Support process on JIRA

...

Service Desk

If the level-0 support fails to answer the user's user´s query, they may then make a request , which is sent to the Copernicus User Support (CUS) Service team at ECMWF. These (level-1) requests will be handled within 8 hours.

For any scientific and specialist enquiries that cannot be answered by the CUS team at ECMWF or addressed by the Copernicus Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).

Enquiries forwarded to the Copernicus User Support Specialist team at DWD will be acknowledged within 3 working days (target 100%) and a notification sent to the user. This step is handled on the JIRA

...

JIRA : Project management software tool created by Atlassian company

...

. Service Desk established at ECMWF for the current project. In case of specific scientific issues, the enquiries will be channeled to the ECV and data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In case of brokered datasets, the data specialist may choose to contact the responsible team at the provider level if they cannot resolve the issue on their own. In each yearly quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks feedback by users, with "1" (very unsatisfied) to "5" (very satisfied). We will also list the number of tickets raised by users on the JIRA Jira system in the Quarterly Report. display-footnotes.

Related articles