Last modified on Jun 27, 2024 17:29

Contributors: Lin Gilbert (University of Leeds), Sebastian B. Simonsen (Technical University of Denmark), Jan Wuite (ENVEO IT GmbH)

Issued by: University of Leeds / Lin Gilbert

Date: 31/05/2020

Ref: C3S_312b_Lot4_D2.IS.2-v2.0_202005_Product_Quality_Assessment_Report_v1.0

Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC2

Note: This document provides the following three deliverables:

            D2.IS.2-v2.0 Product Quality Assessment Report – Ice Velocity

            D2.IS.4-v2.0 Product Quality Assessment Report – Gravimetric Mass Balance

            D2.IS.6-v2.0 Product Quality Assessment Report – Surface Elevation Change

The Gravimetric Mass Balance product was brokered as a whole at the beginning of the project, so its validation has not changed. It is repeated here from the v1 document, C3S_D312b_Lot4.D2.IS.2-v1.0_PQAR_v1.4, for completeness.

History of modifications

List of datasets covered by this document

Related documents

Acronyms

Scope of the document

This document is the Product Quality Assessment Report for the Copernicus Ice Sheets and Ice Shelves service. It presents results of the quality assessment for the provided datasets and a discussion of how well Global Climate Observing System (GCOS) and user requirements have been met.

Executive summary

The service addresses three essential climate variables (ECVs) by providing four separate products.

• Ice velocity is given for Greenland in product D3.IS.4
• Gravimetric mass balance is given for Greenland in product D3.IS.5
• Surface elevation change is given for
  • Antarctica in product D3.IS.6.1
  • Greenland in product D3.IS.6.2

We provide the results from the CDR v2.0 for the ice velocity and surface elevation change datasets produced by the service. The gravimetric mass balance dataset was brokered as a whole at the beginning of the project and has not changed since v1. Because of that, only Section 4 is applicable to the gravimetric mass balance dataset, and this has not changed since v1.0 of this document. It is repeated below for completeness.

This document provides the following three deliverables: Product Quality Assessment Report (PQAR) for Ice Sheets and Shelves Ice Velocity (D2.IS.2-v2.0); PQAR for Gravimetric Mass Balance (D2.IS.4-v2.0); PQAR for Surface Elevation Change (D2.IS.6-v2.0).

1. Product validation methodology

1.1. Greenland ice sheet velocity – D3.IS.4

The ice velocity (IV) product assessment referred to in this document concerns the annually-averaged IV maps of Greenland derived from Sentinel-1 SAR data acquired from 2017-10-01 to 2018-09-30 (reprocessed from CDR v1) and 2018-10-01 to 2019-09-30 (both combined in CDR v2). The quality assessment for ice velocity includes detailed validation with contemporaneous in-situ GPS data at various sites across the ice sheet (Figure 1) and acquired by field teams of the Danish Programme for Monitoring of the Greenland Ice Sheet (PROMICE; Fausto and Van As, 2019) operated by GEUS in collaboration with DTU Space and Asiaq and the Institute for Marine and Atmospheric Research at the University of Utrecht, The Netherlands (IMAU; C.H. Tijm-Reijmer, Pers. Comm).

Figure 1: Greenland Ice Sheet velocity showing the locations of the GPS stations used for validation (red stars: Promice; green stars: IMAU).

The products are also evaluated, on a pixel-by-pixel basis, against publicly available products covering Greenland. These IV maps were produced as part of the NASA 'Making Earth System Data Records for Use in Research Environments' (MEaSUREs) program. The intercomparison provides a good level of quality assurance, in particular in areas where little change is to be expected. For the product intercomparison both the annually averaged map as well as the individual (6/12-day repeat) ice velocity maps, on which the annual maps are based, are considered. Although the individual maps are not provided as a product in C3S, the results are included here as an extra quality metric.

Additionally, we checked the performance of the algorithm in stable terrain, i.e. where no velocity is expected, providing a good overall indication for the bias introduced by the end-to-end velocity retrieval including co-registration of images, velocity retrieval, etc.

Further details on the validation data sets and methodology can be found in the Product Quality Assurance Document.

1.2. Gravimetric Mass Balance D3.IS.5

The Product Validation and Intercomparison Report (PVIR) (Forsberg et al, 2018) provided by the producers of the Gravimetric Mass Balance product (ESA CCI Greenland Ice Sheet (GIS) Essential Climate Variable) provide a full description of the intercomparison activities undertaken to assess the GMB product quality.

1.3. Surface elevation change, Antarctica – D3.IS.6.1

The product is validated against data provided by the Airborne Topographic Mapper (ATM), a scanning laser altimeter flown aboard aircraft by Operation IceBridge (Studinger 2014). A comparison is made between the level 4 product, IceBridge ATM L4 Surface Elevation Rate of Change V001 and a dataset matching its measurements in location and time, calculated from the Antarctic surface elevation change product's underlying data. For further details, see the related Product Quality Assurance Document where both the IceBridge dataset and the validation methodology are described.

As well as validation against external data, to comply with user requirements gathered by the Global Climate Observing System (GCOS) the product should achieve two statistical targets, and the C3S project has identified two key performance indicators, as shown in Table 1.

Table 1: Antarctic surface elevation change product targets

The stability is taken as one standard deviation of the linear model fit to the surface elevation change timeseries used in deriving the surface elevation change rate.

The accuracy is the total error budget for the surface elevation change rate, which has three components - the input uncertainty, the uncertainty due to cross-calibration (if applicable) and the stability, as summarised below.

The input uncertainty is the standard deviation of the surface elevation change values within the aggregation area, i.e. within a single pixel or within a basin.

The uncertainty due to cross-calibration is the standard deviation of the cross-calibration bias estimate. Since v1, the v2 cross-calibration method has been updated to a multiple regressor, using the REGRESS function in the IDL v8 software package. This provides uncertainty estimates for each coefficient in the regression, given input uncertainties as described above.

The three components are summed in quadrature to produce the total error. This method is applicable at both basin and pixel level.

The product only supplies pixel level data. It can be used to make basin level datasets, but the method employed should be selected by the user to best match their needs. The basin level datasets used to check against the targets are made with a simple methodology, where data gaps in any basin at a given time are filled by an ice-velocity-guided value derived from the rest of the basin, using the Berkeley – Ice Sheet Initiative for Climate Extremes (BISICLES) ice velocity model. This, along with a more detailed discussion on the uncertainty processing, is discussed in the Algorithm Theoretical Basis Document.

1.4. Surface elevation change, Greenland – D3.IS.6.2

In line with the Antarctica surface elevation changes, the Greenland counterparts are also validated against data provided by NASA's Operation IceBridge (OIB) Airborne Topographic Mapper (ATM), which is a scanning laser altimeter (Studinger 2014). The ATM instrument has been flown over Greenland since 1993, and within the OIB data-package are estimates of surface change from all repeat measurements of surface elevation done by OIB. This higher-level product (level-4) has been downloaded from NSIDC (https://nsidc.org/icebridge/portal/map). In validation of the surface elevation change (SEC), we use the derived parameter of the cumulative surface elevation change (dh), as this product can be used to derive SEC averaging of the same time span as the time span in-between repeat observations by the OIB ATM. As the OIB data usually are obtained once-a-year (in the spring), we here summarise the mean and standard deviation of the difference in SEC between OIB and our SEC. This mean and standard deviation of the estimate difference is ascribed to the year of the first ATM observation.

As well as validation against external data, to comply with user requirements gathered by the Global Climate Observing System (GCOS) the product should achieve two statistical targets, and the C3S project has identified two KPIs, as shown in Table 1 for Antarctica. The Greenland surface elevation change also uses the KPIs outlined in Table 1, except for the surface coverage which due to more optimal location of the Greenland ice sheet in relation to the satellite orbits and differences in postprocessing interpolation becomes obsolete.

Here, the stability is taken as one standard deviation of the linear model fit to the surface elevation change timeseries used in deriving the surface elevation change rate, whereas the accuracy is the total error budget for the surface elevation change rate. This, along with a more detailed discussion of the uncertainty processing, is discussed in the Algorithm Theoretical Basis Document.

2. Validation results

2.1. Greenland ice velocity – D3.IS.4

Figure 2 shows the results of the intercomparison of Sentinel-1 derived velocity with in-situ GPS measurements for 2017/18 and 2018/19 (CDR v2). The figures show a very good agreement between the GPS and the Sentinel-1 surface velocity. For 2017/18, a total number of 20 stations could be used, showing a mean difference of 2 cm/d and an RMSE of 5 cm/d for the annually averaged ice velocity. For 2018/19, 16 stations could be used showing a mean difference of 3 cm/d and an RMSE of 5 cm/d. Differences can partly be attributed to uncertainties inherent to both methods including differences in spatial sampling: GPS measures a point location, while the feature tracking procedure averages an area for which the size is based on the window size used for image correlation.

Figure 2: Scatter plots (left) and histograms of differences (right) between GPS and Sentinel-1 ice velocity for 2017-2018 (top) and 2018-2019 (bottom).

Excluding the TerraSAR-X derived IV maps that do not fall within the desired temporal range (max time difference of 2 days in comparison to S1 IV maps) leaves a total number of 192, for 2017-2018, and 267, for 2018-2019, usable TSX IV maps for the intercomparison. For each TerraSAR-X derived IV map multiple intercomparisons are possible as the area can be overlapped by multiple S1 tracks over the 2-day time range. In total, 718 (2017/18) respectively 1038 (2018/19) S1 maps fulfill the 2-day criterium and have overlap. For these maps the residuals and their statistics are calculated. Based on a sample size of, combined, more than 11 respectively 9.6 million pixels (for 2017/18 & 2018/19), the overall mean bias between the data sets is well below 1 cm/d for both v_x and v_y components with an RMSE of 0.18 m/d and 0.21 m/d for v_x and v_y respectively.Figure 3 visualises the comparisons and statistical results for the easting (v_x) and northing (v_y) components in a histogram of the residuals.

Figure 3: Top: Histogram of easting (left) and northing velocity (right) residuals of the intercomparison with MEaSUREs TerraSAR-X derived IV maps (selected outlet glaciers) acquired within 2 days of Sentinel-1 derived IV maps for 2017-2018. Bottom: same for 2018-2019.

Figure 4 shows the intercomparison results of the S1 derived ice sheet wide IV map from C3S (2017/18) and MEaSUREs. Based on a sample size of >8 million pixels, the overall mean bias between the two data sets is less than 1 mm/d with an RMSE less than 5 cm/d for both v_x and v_y.

Figure 4: Histogram of easting (left) and northing velocity (right) residuals of the intercomparison with Greenland Ice Sheet velocity map (2017/18).

Differences between the two datasets can have a variety of causes, among others different resolution of the SAR data (TSX vs. Sentinel-1), different temporal range, different settings used for IV retrieval (e.g. matching window, correlation threshold), differences in post processing (e.g. outlier removal, gap filling), different land/ocean and lay-over masks or actual short term velocity fluctuations. In general, higher resolution satellite data captures velocity better, in particular in shear zones, where the velocity gradient is high. The drawback is that often much smaller regions are covered.

Figure 5 shows the results of the stable terrain test as histograms of easting and northing velocity for both annual maps (CDR v2). Based on more than 1.2 million pixels, the outcome of the stable ground test indicates for both products a mean of <0.001 m/d and an RMSE < 0.01 m/d for both easting and northing velocity components.

Figure 5: Top: Histogram of easting (left) and northing (right) velocity in stable terrain for 2017-2018. Bottom: same for 2018-2019.

Table 2 and Table 3 provide a statistical overview of all intercomparison results. For further details, please see related document, the Product Quality Assurance Document.

Table 2: Summary of intercomparison results for 2017-2018, CDR v2 (values in m/d).

Table 3 Summary of intercomparison results for 2018-2019, CDR v2 (values in m/d).

2.2. Gravimetric Mass Balance - D3.IS.5

Please refer to Product Validation and Intercomparison Report (PVIR) (Forsberg et al, 2018) for details on the validation results of the GMB product for Greenland.

2.3. Surface elevation change, Antarctica – D3.IS.6.1

Figure 6 shows the IceBridge comparison results. All IceBridge Antarctic data available from the portal, as of 20^th April 2020, were considered. However, for operation reasons, IceBridge flights were concentrated on the West Antarctic Ice Sheet, Filchner-Ronne Ice Shelf and surrounding regions. In total, 206,404 points of comparison, covering 480 grid cells, were used. Figure 6 shows the locations for which comparison was possible, a scattergram of the values compared and a histogram of their differences. In general, the points of the scattergram lie along or close to the line of equivalence, where Y = X, as expected. The cell-averaged resistant mean difference (i.e. excluding outliers of more than 3 sigma) was 0.081 ± 0.717 m/yr, with a correlation coefficient of 0.55. Although no specific validation target figures were given, this is within the GCOS surface elevation change target accuracy of 0.1 m/yr.

Figure 6: AIS SEC Comparison to IceBridge. Left to right - comparison locations, scattergram of surface elevation change rates for the same locations from IceBridge and the Antarctic SEC products, and histogram of surface elevation change rate differences between the two products.

Figure 7 shows the statistical results. The histograms of dataset accuracy show both component and total figures, and it can be seen that the dominant uncertainty comes from the altimetry measurements. At pixel level the observations are closely clustered, but at basin level they incorporate a wide range of terrain and thus the overall uncertainty is higher.

Coverage changes depend on altimetry mission. ERS-1, ERS-2, Envisat and Sentinel-3A are unable to observe within 8.5° of the South Pole, so 20% of the Antarctic Ice Sheet cannot be covered by these missions. However, they all follow repeating orbits, which allows for regular crossover analysis across the observed regions. CryoSat-2 can observe within 2° of the pole, excluding only 1% of the AIS, but has a drifting orbit on a very long repeat cycle. This makes crossover analysis more challenging, especially in coastal regions where surface slope is high and altimetric measurements are not always possible.

It should be noted that the current release of Sentinel-3A data has known problems when the satellite track crosses from ocean to land, leading to loss of data in coastal regions. Because of this, the Sentinel-3A dataset has less coverage than expected. This problem will be fixed with a new land-ice specific processor which is being run over the mission data from the start, with a data release due in 2020.

The updated cross-correlation method, new to v2, has widened the area available for comparison with Operation IceBridge considerably, almost doubling the number of grid cells available, by retrieving extra data. However, the newly retrieved data tends to be in regions where measurements were noisier or absent, compared to v1. This leads to a wider accuracy distribution. If the v2 method is applied to the cells observed using v1, accuracy between the two methods is very similar – see section 4.3 of the related Product Quality Assurance Document for further discussion.

Figure 7: Statistical summary from Antarctic surface elevation change product. Top to bottom - pixel-level accuracy, basin-level accuracy and percentage coverage of the Antarctic ice sheets and shelves region.

2.4. Surface elevation change, Greenland – D3.IS.6.2

Figure 8 shows the OIB ATM intercomparison results. Here, all Greenland OIB elevation change data available from the NISDC, were considered. The figure shows the geolocated difference between the OIB and C3S observations (right panel), the point-to-point intercomparison and the distribution of the differences. The version 2 solution shows improvements in relation to the GCOS requirements, as we observe a slight increase in the observations with a difference less than 0.1 m/y (vers1: 54.5% vs. vers2: 56.6%). The majority of the OIB observations are located at the fast-changing outlet glaciers, which bias the statistics as the radar observations applied in the C3S elevation change estimates struggle to resolve the short length-scales at which the elevation changes at the outlet glaciers. The mean-difference of all observations is, however, below 0.1 m/year.

Figure 8: (Right) The spatial distribution of the point-to-point intercomparison with the OIB ATM surface elevation estimates. (left-upper) The point-to-point correlations between the C3S-surface elevation change and the OIB surface elevation estimate. (left-lower) The histogram of the point-to-point differences between C3S and OIB surface elevation change. For reference, the C3S v.1 estimate is also included in the left panel.

Figure 9 shows the statistics from the internal-fitting accuracy. Here we see a clear improvement with
the introduction of version 2, with an increased accuracy of more than 25%.

 
Figure 9: The stability in the linear model fit to the surface elevation change timeseries used in deriving the surface elevation change rate. For reference, the C3S v.1 estimate is also included. As seen are the v.2 providing a clear improvement in relation to the GCOS requirements. 

3. Application(s) specific assessments

3.1. Greenland ice velocity – D3.IS.4

Not applicable.

3.2. Gravimetric Mass Balance - D3.IS.5

Not applicable.

3.3. Surface elevation change, Antarctica – D3.IS.6.1

Not applicable.

3.4. Surface elevation change, Greenland – D3.IS.6.2

Not applicable

4. Compliance with user requirements

4.1. Greenland ice velocity – D3.IS.4

The GCOS requirements for accuracy/uncertainty of ice velocity are listed in Table 4. The uncertainty and stability requirements are both 0.1 m/y, well below 1 mm/day, which is unrealistic with present day technology. The accuracy requirements for IV, as described in the User Requirements Document (URD) of the Ice Sheets CCI project (Hvidberg, et al, 2012), identified through an extensive user survey within the glaciology community, lists a minimum accuracy of 30-100 m/y (0.08-0.27 m/d) with an optimum accuracy of 10-30 m/y (0.03-0.08). The results of our quality assessments, all showing cell-averaged mean differences of less than 3 cm/d, fall well within this, more realistic, range.

Table 4: GCOS target requirements for ice sheet velocity (source: GCOS Implementation Plan, 2016)

4.2. Gravimetric mass balance – D3.IS.5

The GRACE solution provided for the major drainage basins are brokered from the Greenland and the Antarctic ice sheet CCI projects. For both processing algorithms and uncertainty estimates we refer to Barletta, Sørensen and Forsberg (2013) and Groh and Horwath (2016).

The primary GCOS, and ESA CCI user requirements (Forsberg et al 2017) for Gravimetric mass balance are met in terms of horizontal resolution, record length and revisit times, as shown in Table 5. If typical ice densities are assumed, the measurement uncertainties are at present about twice the requirement. This emphasises the outstanding scientific question of how to deal with the signal leakages between changing bodies of mass, such as individual drainage basins and peripheral glaciers and ice caps.

Table 5: GCOS target requirements for Ice Mass Change (source: GCOS Implementation Plan, 2016) in comparison to C3S GMB product (Green shows current status, yellow shows planned).

4.3. Surface elevation change, Antarctica – D3.IS.6.1

In validation, the cell-averaged mean difference (excluding outliers beyond 3 standard deviations) between the independent and C3S datasets was 0.081 ± 0.717 m/yr. This is within the GCOS target accuracy of 0.1 m/yr.

In accuracy, at pixel level the C3S target of 0.1 m/yr is reached or bettered in approximately one third of all instances, with the peak of the distribution lower than the target, at 0.075m. At basin level the additional uncertainty from the larger collection area and data gap filling means that the majority of instances do not reach the GCOS target. The distribution is wide and flat, and peaks close to the target at 0.155m.

In stability, the GCOS target of 0.1 m/yr is reached or bettered in approximately 82% of all instances.

In coverage, the C3S target of 65% is generally achieved during the ERS-2 and Envisat mission periods, but during the ERS-1 period the coverage is generally 20% lower due to sparse altimetric data. During the CryoSat-2 mission period the C3S target of 90% is never achieved, in fact only approximately half of that is observed in any given month due to the drifting orbit as described previously in Section 2.3. As also described in that section, Sentinel-3A coverage is less than it should be, which will be fixed in the next data reprocessing, but it has coverage above 60%, just below its 65% target.

4.4. Surface elevation change, Greenland – D3.IS.6.2

The uncertainty given in the product is the epoch uncertainty (derived from the supplied input data) and the model uncertainty (derived from the plane-fitting) combined. The evaluation of the uncertainty estimates is shown in Figure 8 and Figure 9. As seen, the number of points with an accuracy within the GCOS requirements are 97.8%. This estimate is purely an internal stability indication and not the real error estimate, as multiple factors are not included in this estimate. Factors such as changes in penetration depth of the radar, slope induced relocation errors and that the radar only observers changes at the highest point within its footprint, which may bias the elevation change estimate. Therefore, the real error estimate and the number which needs to fulfill the user-requirements needs to be found by applying independent validation of the surface elevation change product. Here, the independent validation is provided by comparison to NASA's OIB airborne laser altimetry campaigns. Figure 8 shows the result of the inter-comparison between the OIB and C3S-Greenland SEC, with the mean bias of all observations of less than 0.1 m/yr. Thereby, the current processing version (C3S_GrIS_RA_SEC_25km_vers2_2020-04-17.nc as of the date of this document) fulfils the GCOS requirement. The spatial coverage of the OIB data is limited and Figure 9 shows the annual-distribution of grid-cell accuracy as given in the product. The OIB dataset is referenced in the Algorithm Theoretical Basis Document.

References

Barletta, V.R., Sørensen, L.S. and Forsberg, R. (2013). Scatter of mass changes estimates at basin scale for Greenland and Antarctica. The Cryosphere, 7, 1411-1432

Fausto, R.S. and van As, D., (2019). Programme for monitoring of the Greenland ice sheet (PROMICE): Automatic weather station data. Version: v03, Dataset published via Geological Survey of Denmark and Greenland. DOI: https://doi.org/10.22008/promice/data/aws
Forsberg, R. et al.(2017) User Requirements Document (URD), ESA Climate Change Initiative (CCI), Greenland Ice Sheet (GIS) Essential Climate Variable (ECV), Version, 2.4, 22-11-2017, ST-DTU-ESA-GISCCI-URD-001, available at: http://esa-icesheets-greenland-cci.org/index.php?q=webfm_send/169

Forsberg, R. et al. (2018) Product Validation and Intercomparison Report (PVIR), ESA Climate Change Initiative (CCI), Greenland Ice Sheet (GIS) Essential Climate Variable (ECV), Version, 3.0, 25-06-2018, ST-DTU-ESA-GISCCI-PVIR-001, available at: http://esa-icesheets-greenland-cci.org/index.php?q=webfm_send/180

Groh, A., and Horwath, M. (2016). The method of tailored sensitivity kernels for GRACE mass change estimates. Geophysical Research Abstracts, 18, EGU2016-12065.

Hvidberg, C.S., et al., User Requirements Document for the Ice_Sheets_cci project of ESA's Climate Change Initiative, version 1.5, 03 Aug 2012. Available from: http://www.esa-icesheets-cci.org/

Studinger, M. (2014). IceBridge ATM L4 Surface Elevation Rate of Change, Version 299 1, Antarctica subset. N. S. a. I. D. C. D. A. A. Center. Boulder, Colorado, USA. DOI: 10.5067/BCW6CI3TXOCY

5. Related articles

• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Product User Guide Specification (PUGS)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: System Quality Assurance Document (SQAD)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Product Quality Assurance Document (PQAD)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Product Quality Assessment Report (PQAR)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Algorithm Theoretical Basis Document (ATBD)