Contributors: S. B. Simonsen (Technical University of Denmark)
Issued by: Technical University of Denmark / S. B. Simonsen
Date: 06/03/2024
Ref: C3S2_312a_Lot4.WP3-SQAD-IS-v2_202401_SEC_System_Quality_Assurance_v5_i1.0
Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
General definitions
Baseline: A combination of processor versions, auxiliary data and other needed enablers that allows the generation of a coherent set of Earth observation products.
Climate Data Record (CDR): A time series of measurements of sufficient length, consistency and continuity to determine climate variability and change.
Crossover analysis: A method for deriving elevation change at locations where the orbits of a single or multiple satellites cross.
Cycle: A satellite's cycle is one full completion of its track over the ground, after which the ground track repeats.
Processing chain: A sequence of software routines run to convert input data to an output product.
Processing level: European Space Agency (ESA) datasets are labelled by the level of processing applied to them; level 0 corresponds to raw data, level 1 to acquired data in physical units, and level 2 to values of the parameter constituting the scientific objective of the experiment. Higher levels involve extrapolation or assimilation. Occasionally intermediate levels are produced, e.g., CryoSat-2 has a L2i product, which is intermediate between L2 and L3.
Stability: An estimate of the consistency of the measurements over time.
Surface Elevation Change (SEC): The surface elevation of a point on an ice sheet is the height of the ice sheet surface above a reference geoid (a hypothetical solid figure whose surface corresponds to mean sea level and its imagined extension under land areas). An increase in surface elevation over time at a given location indicates a gain of ice or snow at that location, and conversely, a decrease indicates a loss. The surface elevation change product provides the rate of change given at monthly intervals at each location on a grid covering the ice sheet. The definition of the grid projection includes the geoid used. Given the rates of change, absolute change can be calculated for any time period.
Validation: Comparison between two independent datasets to test their agreement.
Scope of the document
This document is the System Quality Assurance Document (SQAD) for Surface Elevation Change (SEC) as part of the Copernicus Ice Sheets and Ice Shelves service. The latest version, Version 5.0, includes updates solely for the Greenland data product, while the production of Antarctic data has been temporarily halted. We refer the reader to SQAD version 4.0 [D3] for a description of the Antarctic SEC, also hosted at the Copernicus Climate Data Store (CDS). This document describes the organisation of the data processing system and how product updates are implemented. This issue is based on the production and support systems for the v5 products for Greenland SEC CDR made by the service.
Executive summary
This document describes the computing resources and processes involved in making the surface elevation change data products for Greenland.
The first section gives an overview of inputs, processing chains and outputs, for product generation and validation. It details the input radar altimeter datasets, the validation datasets used in quality control, and the auxiliary datasets needed during the processing. High-level descriptions and a flowchart of the processing chains are given. The computing resources available to the service team are discussed.
The product is updated on two different timescales. A Climate Data Record (CDR) is produced annually and may incorporate an updated version of an input dataset, or improvements in any processing chain. Over the following year, intermediate CDRs (ICDRs) are produced monthly to incorporate recent input data, but the underlying methodology and input products are not changed.
The second section describes how the ICDRs are produced. New data is assimilated monthly, taking all full cycles of data that finish within the monthly period. Different input datasets have different cycle times.
The third section describes how the CDRs are processed, and how to regenerate products in case of failure. CDR processing happens annually, and validation is also run annually, to match the update cycle of the validation datasets.
The fourth section describes data storage and computing system backups, and mitigation plans in the event of a major systems failure.
The fifth section describes arrangements for user support.
1. System Overview
1.1. System elements and interfaces
The processing system consists of
- one ingestion chain per input dataset
- one chain to combine the ingested data into the final product
- two validation chains
A flowchart of the processing chains is shown in Figure 1.1.
The amount of data needed for processing depends on the satellite mission. While 5 years of data are used for older satellites (ERS-1, ERS-2, and Envisat), ongoing missions (CryoSat-2, and Sentinel-3) use only 3 years of data. The input datasets used for CDR v5 are European Remote Sensing satellite 1 (ERS1 Phase C REprocessing of Altimeter Products for ERS (REAPER) L2, ERS1 Phase G REAPER L2, ERS2 REAPER L2, Envisat Geophysical Data Record (GDR) v3, CryoSat-2 L2i baseline D, Sentinel-3A L2 and Sentinel-3B L2 (same input data as for WP2-FDDP-SEC-CDR-AntIS-v4, see Table 1.1).
Table 1.1. Input datasets to ingestion processing chains.
Satellite | Sensor | Dataset name | Processing level | Website | Reference |
|---|---|---|---|---|---|
ERS1 phase C | RA | REAPER | L2 | https://earth.esa.int/eogateway/activities/reaper?text=reaper | Brockley et al, 2017 |
ERS1 phase G | RA | REAPER | L2 | As above | Brockley et al, 2017 |
ERS2 | RA | REAPER | L2 | As above | Brockley et al, 2017 |
Envisat | RA-2 | GDR v3 | L2 | https://earth.esa.int/web/guest/-/ra-2-geophysical-data-record-1470 | Femenias (ed), 2018 |
CryoSat-2 | SIRAL | Baseline D to August 2021, baseline E thereafter1 The two baselines are used in sequence because neither is available for the full mission at time of CDR production. They differ in only minor ways, so this usage is possible. | L2i | https://earth.esa.int/web/guest/-/how-to-access-cryosat-data-6842 | CryoSat-2 Product Handbook, 2019 |
Sentinel-3A | SRAL | SR_2_LAN_NT | L2 | ACRI-ST IPF Team, 2020 | |
Sentinel-3B | SRAL | SR_2_LAN_NT | L2 | As above | ACRI-ST IPF Team, 2020 |
For v5 the Sentinel-3 Thematic product has been included in the processing. A flowchart of the processing chains is shown in Figure 1.1.
Figure 1.1. High-level flow chart of the Greenland surface elevation rate processing chains. Left: Older mission processing, middle: the flow chart for the newer satellites and the combined mission processing, right: validation.
As data is only ingested upon need, the ERS1, ERS2, and Envisat ingestion only has to be run once. CryoSat-2 and Sentinel-3 data, on the other hand, will be ingested monthly to produce the newest ICDRs. The processing of ERS1, ERS2, and Envisat relies on the optimal combination of the plane-fitting (PF)3 and repeat-track (RT) algorithms. The drifting orbit of CryoSat-2 does not allow for RT solutions; thus, the combined CryoSat-2 and Sentinel-3 (as well as CryoSat-2 alone) surface elevation change is derived only by the PF algorithm. The processing chain performs mission cross-calibration as a part of the implemented processing algorithm. For more details, see Section 2.3 of the related Algorithm Theoretical Basis Document (ATBD) [D1].
The ingestion chain for CryoSat-2 and Sentinel-3A/B will run monthly as long as CryoSat-2 and/or Sentinel-3 is operational. All intermediate datasets are archived. The final processing chain merges these files into a single dataset using kriging to average intermediate monthly elevation rates closest in time to the final-solution timestamp. Finally, key performance indicators are derived before product output. The performance indicators are the product accuracy, stability, geographical coverage, and percentage of data points within a given uncertainty. Following these performance indicators, the validation chain is run annually. It compares the derived surface elevation change rates to those from Operation IceBridge (see Studinger 2014) and surface elevation changes from the ICESat-2 ATL15 product. This procedure is described in more detail in the related Product Quality Assurance Document (PQAD) [D2].
All European Space Agency (ESA) radar altimetry data is downloaded to the storage servers at the Technical University of Denmark (DTU) daily. The data storage forms the basis of the operational system developed within ESA's Climate Change Initiative (CCI) and is also the backbone of the process in the Ice Sheets and Ice Shelves service of the Copernicus Climate Change Service (C3S) Land Hydrology and Cryosphere project. All necessary input and auxiliary datasets and models are freely available, although for some registration is required. For more details, including web addresses, see the related Algorithm Theoretical Basis Document [D1], Section 2.2.
1.2. Hardware, supercomputers and cloud computing
The team at DTU Space has access to a scalable range of computing facilities. A High-Performance cluster with 48 threads has been chosen for the Greenland SEC processing. All development has been done on a macOS system with 4 cores and a limited amount of memory and storage capabilities.
2. Upgrade cycle implementation procedure
Both Sentinel-3 and CryoSat-2 level-2 data are usually available for download approximately 35 days after they are acquired by the satellite. The system setup at the Technical University of Denmark downloads all new data daily, and the Greenland surface elevation change processors are run every month by an automated processing procedure. When the automated processing is done, a processing summary is sent to all relevant persons at DTU Space. When the processing e-mail is received, the final product undergoes human inspection before being pushed to a ftp-site, where the product is released for the Climate Data Store (CDS). In practice, this will result in a time lag between data acquisition and product update of about 2 months.
3. Procedures for reprocessing CDR's
All input data and intermediate outputs, as well as the CDRs and ICDRs, are stored. Reprocessing could be achieved by starting from the input data for all missions, but typically only the relevant processing chains are run, e.g., when an upgraded input dataset becomes available for a single mission then the single-mission ingestion is re-run. In practice, reprocessing has always been done to coincide with yearly improvements to the processing system. Between CDRs v2 and v3 both the Envisat and CryoSat-2 products were upgraded. In v5, the Sentinel-3A/B data has been revised to utilize Sentinel thematic products.
The validation dataset from Operation IceBridge, is also updated yearly, but irregularly, with flight campaigns in the Arctic. The final flights were performed in November 2019. Validation is performed yearly to accommodate this schedule, using the full dataset from the project start. This necessitates the re-running of the validation chain from the beginning.
For version 5 the validation against ICESat-2's ATL15 product, which is currently only available for a two-year period, is also made. The validation chain can be re-run if the product is upgraded or its time period is extended.
4. System maintenance and system failures
The system at DTU Space undergoes an incremental backup each day and additionally, full system backup is performed quarterly, both on physical media. System maintenance is kept to short periods and advanced warning is given. Such system maintenance cycles will not affect the product's users as the monthly product update cycle can be planned around known system outages. If needed the processing chain can be run on alternative servers, as the code needed to run the full processing chain is written in python 3.5, allowing the system to run on any replacement systems, which is found suitable in the unlikely event of a major failure. The product will be archived at the Technical University of Denmark, but copies will be pushed to the Earth Observation data Centre (EODC), who will supply them to the CDS. Users can only obtain data via the CDS. Thus, failure of the Technical University of Denmark system will not affect the availability of any product already made. In unlikely circumstances, failure could delay production of further ICDRs, in which case the CDS will be informed.
5. User support
The Ice Sheets and Ice Shelves service has a team account with the Copernicus User Support (CUS) Jira Service Desk System, to provide level 2 user support, i.e. to answer enquiries specific to their products, by direct interaction with the user through the Jira helpdesk.
Once a request is sent, the Copernicus User Support Service team at the European Centre for Medium-Range Weather Forecasts (ECMWF) will handle the requests within 8 hours (level 1).
For any scientific and special enquiries that cannot be answered by the CUS team at ECMWF or addressed to the Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).
Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. In case of specific scientific issues, the enquiries will be channelled to the ECV and data specialist of the C3S2_312a_Lot4 project and should be resolved within 3 working weeks (target 85%). In each quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets in the Quarterly Report.
All products made by the team, including the two Surface Elevation Change products, are handled by the same system.
References
ACRI-ST IPF Team (2020). Product Data Format Specification – SRAL/MWR Level 2 Land products. ESA document reference S3IPF.PDS.003.2. Available from https://sentinel.esa.int/documents/247904/2753172/Sentinel-3-Product-Data-Format-Specification-Level-2-Land (URL resource last accessed 19th December 2023)
Brockley, D. et al. (2017). REAPER: Reprocessing 12 Years of ERS-1 and ERS-2 Altimeters and Microwave Radiometer Data. IEEE TGRS, June 2017. DOI: 10.1109/TGRS.2017.2709343
CryoSat-2 Team (2019). CryoSat-2 Product Handbook, baseline D 1.1. ESA document reference C2-LI-ACS-ESL-5319. Available from
https://earth.esa.int/eogateway/documents/20142/37627/CryoSat-Baseline-D-Product-Handbook.pdf/c76df710-2a5c-c8b8-00c1-13c8db0e9f51 (URL resource last accessed 19th December 2023)
Femenias, P. (editor) (2018). Envisat Altimetry Level 2 Product Handbook. ESA document CLS - ESLF - 18 -0003. Available from https://earth.esa.int/handbooks/ra2-mwr/ (URL resource last accessed 19th December 2023)
Ivins, E. R. and James, T. S., (2005). Antarctic glacial isostatic adjustment: a new assessment. Antarctic Science, 17(4), 541-553
Padman, L., et al (2002). A new tide model for the Antarctic ice shelves and seas. Annals of Glaciology, 34, 247-254
Rignot, E., Mouginot, J. and Scheuch, B. (2011). Ice Flow of the Antarctic Ice Sheet. Science 333 (6048):1427-1430
Slater, T., et al, (2018). A new digital elevation model of Antarctica derived from CryoSat-2 altimetry, The Cryosphere, 12, 1551-1562, doi: 10.5194/tc-12-1551-2018
Smith, B. (2021). ICESat-2 Algorithm Theoretical Basis Document for Land Ice DEM and Land Ice Height Change Release 001 Algorithm Theoretical Basis Document (ATBD) for Land-ice DEM (ATL14) and Land-ice height change (ATL15).
https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL14_ATL15_ATBD_r001.pdf (URL resource last accessed 12th December 2023)
Studinger, M., 2014, updated 2018. IceBridge ATM L4 Surface elevation rate of change, version 1. Boulder, Colorado, USA. NASA Snow and Ice Data Center Distributed Active Archive Centre. DOI: 10.5067/BCW6CI3TXOCY
Zwally, H. Jay, Mario B. Giovinetto, Matthew A. Beckley, and Jack L. Saba, 2012, Antarctic and Greenland Drainage Systems, GSFC Cryospheric Sciences Laboratory at https://earth.gsfc.nasa.gov/cryo/data/polar-altimetry/antarctic-and-greenland-drainage-systems (URL resource last accessed 19th December 2023)
