Contributors: A. Velazquez Blazquez (Royal Meteorological Institute of Belgium (RMIB)), N. Clerbaux (RMIB), E. Baudrez (RMIB), S. Dewitte (RMIB), S. Nevens (RMIB)
Issued by: RMIB
Date: 30/09/2020
Ref: C3S_D312b_Lot1.2.5.1-v2.0_202003_SQAD_ECVEarthRadiationBudget_v1.0
Official reference number service contract: 2018/C3S_312b_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
General definitions
Not applicable
Scope of the document
This document describes the systems that are used to generate the Total Solar Irradiance (TSI) timeseries which is published in the Copernicus Climate Change Service (C3S) Climate Data Store (CDS). It also provides information on the procedures implemented in the event of problems or outages in the systems and their input data, whether planned or unplanned, and information on the provision of user support.
Executive summary
The Total Solar Irradiance (TSI) quantifies the amount of solar energy that is received by the Earth.
TSI is defined as the amount of solar power that reaches the Earth's top of the atmosphere per unit surface area perpendicular to the Sun–Earth direction at the mean Sun–Earth distance.
The TSI timeseries can be accessed via the Copernicus Climate Data Store (CDS) at https://cds.climate.copernicus.eu.
Back-end user support is provided by a point of contact at RMIB (Dr. Steven Dewitte, with copy to N. Clerbaux and A. Velazquez-Blazquez), ensuring a response to product related user queries within the allocated time frame and meeting the relevant KPI targets.
1. System overview
1.1 System elements and interfaces
For the “active” TSI instruments, daily downloads of fresh data is performed using the wget command. These downloads are launched automatically by the Linux cron daemon at 05:00 UTC.
Then, the ICDR is updated to incorporate the new data. The processing is based on the “make” utility program that handles the dependencies between the data files, as specified in a ‘makefile’.
There is an off-line process that must be performed each time a new TSI instrument is added to the processing. This off-line process determines the TSI scaling factor needed to put the data on the so-called Space Absolute Radiometric Reference (SARR) level. This processing is described in the ATBD [D3] and in the journal papers [D1] and [D2].
The C3S TSI TCDR/ICDR are generated and archived at the Royal Meteorological Institute of Belgium (RMIB). The C3S CDS can access the daily TSI timeseries, using simple HTTP or FTP requests, at:
1.2 Hardware and data server system
1.2.1 Computing infrastructure
The data downloads (via wget) and the TCDR and ICDR processings are performed on the RMIB computing server "typhoon". A hot backup of the processing is performed on the backup computing server "tornado". The "typhoon" and "tornado" are physical Linux servers.
1.2.2 Data server system – Input data
For the "active" instruments, the input data are downloaded on a daily basis from the following ftp/http servers:
DIARAD/VIRGO | ftp on the VIRGO Data Center (VDC) |
PMO6/VIRGO | ftp on the VIRGO Data Center (VDC) |
TIM/SORCE | http://lasp.colorado.edu/data/sorce/tsi_data/daily/sorce_tsi_L3_c24h_latest.txt |
TIM/TCTE | http://lasp.colorado.edu/data/tcte/tsi_data/tcte_tsi_L3_c24h_latest.txt |
TIM/TSIS | http://lasp.colorado.edu/data/tsis/tsi_data/tsis_tsi_L3_c24h_latest.txt |
Data sources for the other instruments in the composite are specified in the ATBD [D3] under the “Input Data” Section.
1.2.3 Data server system – Generated composite data
The results are then copied on the RMIB GERB HTTP/FTP server (gerb.oma.be) for data dissemination. The CDS accesses the TSI data on this server. The "gerb.oma.be" HTTP/FTP server is running on a virtual machine.
2. Upgrade cycle implementation procedure
The ICDR is updated automatically each day to incorporate new data, at 05:00 UTC. The resulting TSI timeseries is labeled as ‘latest’:
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_ICDR_v2_latest.txt
There is no versioning in the name of this data file. This stresses the fact that it is generated automatically, without any quality check. For traceability, the date of the processing is included in the header, at beginning of the data file.
Regularly (each 3 months), the “latest” ICDR is validated and then released as new version of the ICDR. There will be three ICDR batches v2.1-v2.3:
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_ICDR_v2.1.txt
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_ICDR_v2.x.txt
(…)
Validation results are included in the PQAR document [D4].
3. Procedures for reprocessing CDR's
The daily TSI TCDR timeseries has been processed twice in the frame of C3S 312 lot1 contract and are available as version v1.0 and version 2.0 at:
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_TCDR_v1.0.txt
https://gerb.oma.be/tsi/C3S_RMIB_daily_TSI_composite_TCDR_v2.0.txt
With respect to the v1.0, the version v2.0 :
- includes new data for 2017 and 2018,
- includes historical data from 1979-1984
- implements temporal interpolation of missing daily data using SATIRE-S data as “proxi” for the interpolation (see ATBD [D3]).
The procedure for the reprocessing toward version 3.0 is not yet defined.
4. System maintenance and system failures
The planned outages of the server, and therefore the availability of data, will be communicated as far as possible to the CDS well in advance. Any notification procedures requested by the CDS will be complied with.
The servers are monitored by the RMIB in the frame of the C3S 312B Lot1 contract. In the case of unforeseen system failures, or other problems affecting data delivery through the HTTP/FTP servers, the CDS will be notified as quickly as possible, following any notification procedures requested by the CDS. For longer outages (longer than a few hours) the CDS will be notified again once the issue has been resolved.
Issues or outages, whether planned or unplanned, will be documented in the Quarterly Report.
Note that the TSI composite processing is running in parallel on 2 physical servers: typhoon (operational server) and tornado (backup server). The likelihood of simultaneous problem is therefore relatively low, although several single-point-of-failures are identified (e.g. network switch, FTP/HTTP virtual machine).
The access to the input data (individual timeseries) is verified before execution of the composite processing. In case of data access problem to one of these input files, the "latest" ICDR is not updated and error messages are sent (via email) to the relevant RMIB scientists for action.
5. User support
5.1 Enquiries
The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base (CKB) (CAMS and C3S Knowledge Base (CKB)) or the Copernicus Climate Data Store desk (https://cds.climate.copernicus.eu/contact-us). This is known as the level-0 process of User Support.
Figure 1: The schematic of Copernicus User Support (Copernicus User Support Handbook, V2.1)
Contact and User Support process on JIRA service desk
If the level-0 support fails to answer the user's query, they may then make a request, which is sent to the Copernicus User Support (CUS) Service team at ECMWF. These (level-1) requests will be handled within 8 hours.
For any scientific and specialist enquiries that cannot be answered by the CUS team at ECMWF or addressed by the Copernicus Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).
Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. This step is handled on the JIRA 1 Service Desk established at ECMWF for the current project. In case of specific scientific issues, the enquiries will be channeled to the ECV and data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In case of brokered datasets, the data specialist may choose to contact the responsible team at the provider level if they cannot resolve the issue on their own. In each yearly quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets raised by users on the JIRA system in the Quarterly Report.
- JIRA: Project management software tool created by Atlassian company ↩
