Contributors: Jacqueline Bannwart (University of Zurich), Inés Dussaillant (University of Zurich), Frank Paul (University of Zurich), Michael Zemp (University of Zurich)
Issued by: UZH / Inés Dussaillant
Date: 26/03/2024
Ref: C3S2_312a_Lot4.WP2-FDDP-GL-v2_202312_MC_PQAR-v5_i1.0
Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
General definitions
Elevation change: Vertical change in glacier surface elevation (altitude), typically derived from two elevation measurements, adjusted if necessary for the difference of their respective datum surfaces, at the same (or nearly the same) horizontal coordinates (Cogley et al., 2011).
Geodetic method: Any method for determining mass balance by repeated mapping of glacier surface elevations to estimate the volume balance; cartographic method and topographic method are synonyms. The conversion of elevation change to mass balance requires information on the density of the mass lost or gained, or an assumption about the time variations in density (Cogley et al., 2011).
Glaciological method: A method of determining mass balance in-situ on the glacier surface by measurements of accumulation and ablation, generally including measurements at stakes and in snow pits; direct method has long been a synonym. The measurements may also rely on depth probing and density sampling of the snow and firn, and coring. They are made at single points, the results from a number of points being extrapolated and integrated to yield the surface mass balance over a larger area such as an elevation band or the entire glacier (Cogley et al., 2011).
Gravimetric (Gravity) method: A technique in which glacier mass variations are calculated from direct measurements of Earth's gravity field. Satellite gravimetry is at present the most feasible method for determining glacier mass balance from changes in gravity. The Gravity Recovery and Climate Experiment (GRACE) consists of two polar-orbiting satellites separated by about 200 km along-track, and is the primary mission for this work to date (Cogley et al., 2011).
Scope of the document
This document is the Product Quality Assessment Report (PQAR) for the glacier change service providing a global gridded annual glacier mass-change product to the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S). It presents the results of the quality assessment and validation of the glacier mass-change product. Furthermore, it describes some additional applications specific assessment performed with the products. It also reviews the quality of the products against the data requirements and provides recommendations on data usage.
Executive summary
This document provides a description of the quality assessment and validation results for the dataset (Climate Data Record; CDR) provided by the C3S Glacier Change Service to the Climate Data Store (CDS). The Glacier Change Service addresses the essential climate variable (ECV) Glacier and provides a globally distributed product of annual glacier mass-changes with a spatial resolution of 0.5° x 0.5° covering the hydrological years from 1975/76 to 2021/22.
The CDS glacier change product is a one-of-a-kind dataset. Since it builds on all available glacier change observations available from the glaciological community through the Fluctuations of Glaciers (FoG) database, there are no independent validated reference datasets to compare it with for accuracy assessment. Therefore, validation and accuracy assessment of the C3S glacier change product is limited to four general levels:
(i) Peer review system of original data publication in an academic journal
(ii) The evaluation of the uncertainty information submitted with the input data
(iii) Leave one out cross validation on reference glaciers
(iv) Comparison with previous global glacier change assessments
Points (i) and (ii) relate to the glaciological and geodetic time series used as input data, and are discussed in World Glacier Monitoring Service (WGMS, (2023)) and in Zemp et al. (2013, 2015, 2019). The uncertainty assessment of the global gridded annual glacier mass-change product CDR is extensively described in RD1. In this document we will focus mainly on points (iii) and (iv). First we provide a brief summary of the leave-one-out cross validation methodology in Section 1, followed by the validation results in Section 2. In Section 3, we present the improvements of the glacier mass-change CDR with respect to previous observational based global glacier change assessments (Zemp et al., 2019, 2020; Hugonnet et al., 2021) and compare it to independent glacier mass change assessment from the Gravity Method (Wouters et al., 2019). Further information on compliance with the user requirements (detailed in RD3) is provided in Section 4.
1. Product validation methodology
This section briefly describes the validation methodology for the global gridded annual glacier mass-change product Climate Data Record (CDR), for more details the reader is referred to the Product Quality Assurance Document (PQAD; RD2). There are no independent measurements available to compare and validate our gridded glacier mass-change product on a direct way, basically due to the fact that the input datasets themselves (FoG elevation and mass change time series) are generally used as validation reference. Therefore, we propose to validate our results with an independent leave-one-out crossed validation experiment (see section 1.2) over selected reference glaciers, i.e. glaciers having already validated observations.
1.1. WGMS Reference Glaciers
With the reference glaciers, the World Glacier Monitoring Service (WGMS) aims at providing a reliable and well-documented sample of globally distributed long-term observation series to document the impact of climate change on glacier mass-balance (Figure 1). The reference glaciers have more than 30 years of ongoing glaciological mass-balance measurements. Thanks to the wealth of data and long-term observation series, the WGMS reference glaciers represent excellent test sites for the validation of glaciological and geodetic data and related production methods. They are used here as the main reference dataset for validation of the C3S global gridded glacier mass-change product CDR.
Figure 1: Global map showing the location of WGMS reference glaciers. Further details can be found on the WGMS reference glacier website1. Data sources: Reference glaciers data from WGMS. Mountains above 1000m (brown shades) from GTOPO30 digital elevation model from US Geological Survey (USGS).
1.2. Leave one out cross validation
We use 32 reference glaciers for this assessment. For each reference glacier we compare the reference validated mass balance time series, available directly from the FoG database, with the leave-one-out calibrated mass balance series calculated using our methodology (described in detail in RD1).
For the latter, following our original method, we calculate the reference glacier calibrated mass balance series by calibrating the spatial anomaly over the glacier geodetic sample. For the sake of the experiment, the spatial anomaly in this case is obtained as the mean of nearby individual glacier annual anomalies selected by the special search excluding (leaving out) the selected reference glacier from the glaciological sample. The Root Mean Square Error (RMSE) is calculated as a measure of the deviation of the calculated observations from the reference sample, and the r correlation factor as a measure of the correlation between the reference and calculated leave-one-out cross validation time series. The RMSE resulting from this comparison will provide further insight on the validity of our individual glacier accuracy and uncertainty assessment.
2. Validation results
Validation results are shown for all 32 reference glaciers together (Figure 2), and for three individual reference glaciers located in different regions: Hintereisferner, Aalfotbreen and Gulkana glaciers (Figures 3a, 3b, 3c). Considering all 32 reference glaciers together, we find a mean deviation (RMSE) of 0.55 m w.e. with respect to the reference sample.
Hintereisferner is the best case of the three individual glaciers presented, with a RMSE of 0.3 m w.e. and a correlation (r) of 0.93, stable between past and present periods (past period before year 2000 has fewer geodetic estimates; present period after year 2000 is better represented by the geodetic sample). Aalfotbreen and Gulkana presents a larger RMSE of 0.8 m w.e. The correlation between the reference and calculated mass balance series is better for Aalfotbreen (r = 0.93) than for Gulkana (r = 0.45). We found a consistent underestimation of the calculated mass balance extreme values for almost all the validation reference glaciers (where reference mass balance are more negative than the calculated estimates for the negative values and more positive than calculated estimates for the positive values). This effect is clearly represented by the slope of the regression fit with respect to the 1:1 line in Figures 2, 3b and 3c. This bias is accounted for in the uncertainty assessment as discussed in the following paragraphs, however we note that there is still room for improvement of the algorithm in this particular aspect.
Figure 4 shows the analysis results of the leave-one-out cross validation for the previous glacier change product version WGMS-FOG-2022-09. We compared, for each reference glacier, the cross validation RMSE (i.e. mean deviation of the calculated annual mass balance (Ba) values with respect to the reference Ba values) against the calibrated estimate uncertainty estimated by our original methodology using the Standard Deviation as measure of the error. These results show that the estimated error was in average 682 mm w.e. (i.e 0.68 m w.e.) larger considering all 32 glaciers in the validation sample, confirming that our uncertainty assessment was too conservative and may be reduced.
Justified by the previous analysis, we use the Standard Error as a measure of the uncertainties for the new glacier change product version WGMS-FOG-2023-09, and performed the same analysis. The new results (Figure 5) show that the calculated error is on average 0.3 m w.e. larger than the cross validation RMSE, confirming that our uncertainties are still within an acceptable range and able to capture all sources of random and systematic errors. By changing to the Standard Error as measure of uncertainty we were able to both (i) reduce uncertainties and (ii) produce more realistic uncertainties by accounting for the number of observations (i.e. years with less observations get larger errors than years having a larger observation sample).
The leave-one-out cross validation results prove that our algorithm is able to correctly capture the annual variability of individual glacier mass balances on glaciers not having glaciological time series (99% of the global glaciers).
Figure 2: Leave -one-out cross validation results for all 32 reference glaciers. The individual reference glaciers annual mass balance (Ba) observations are plotted against the leave one out calculated annual mass balance using our methodology. Every dot corresponds to a yearly observation. The RMSE shows a mean deviation of 0.55 m w.e. of the calculated sample with respect to the reference sample.
Figure 3 a. Example reference glacier Hintereisferner, located in Central Europe (Randolph Glacier Inventory – RGI60 – 1st order region 11, CEU). Hintereisferner reference annual mass balance observations against the leave one out calculated annual mass balance using our methodology. Every dot corresponds to a yearly observation. Nearby glaciers selected to calculate the spatial anomaly are listed together with their distance to Hintereisferner..
Figure 3 b. Example reference glacier Aalfotbreen, located in Scandinavia (RGI60 1st order region 8, SCA). Aalfotbreen reference annual mass balance observations against the leave one out calculated annual mass balance using our methodology. Every dot corresponds to yearly observations. Nearby glaciers selected to calculate the spatial anomaly are listed together with their distance to Aalfotbreen.
Figure 3 c: Example reference glacier Gulkana, located in Alaska (RGI60 1st order region 1, ALA). Gulkana reference annual mass balance observations against the leave one out calculated annual mass balance using our methodology. Every dot corresponds to yearly observations. Nearby glaciers selected to calculate the spatial anomaly are listed together with their distance to Gulkana.
Figure 4: Leave-one-out cross validation analysis for glacier change product version WGMS-FOG-2022-09. Reference glacier leave-one-out cross validation RMSE against the estimated uncertainty (Standard Deviation) of the calibrated estimate calculated by our methodology for the same glacier. Each symbol corresponds to one of the 32 reference glaciers used for cross validation. The size of the symbol corresponds to the Area of the reference glacier. The RMSE of this graph shows that the calculated sigma is on average 682 mm w.e. (i.e 0.68 m w.e.) larger than the cross validation RMSE, confirming that our uncertainty estimation was too conservative.
Figure 5: Leave-one-out cross validation analysis for glacier change product version WGMS-FOG-2023-09. Reference glacier leave-one-out cross validation RMSE against the estimated uncertainty (Standard Error) of the calibrated estimate calculated by our methodology for the same glacier. Each symbol corresponds to one of the 32 reference glaciers used for cross validation. The size of the symbol corresponds to the Area of the reference glacier. The RMSE of these results show that the calculated error is on average 0.3 m w.e. (i.e 300 mm w.e.) larger than the cross validation RMSE, confirming that our uncertainty estimation is able to capture errors within an acceptable range.
3. Application(s) specific assessments
3.1. Comparison to previous observation based global glacier mass change assessments
An alternative way to validate and assess the accuracy of our glacier change product is by direct comparison with previous global observation based glacier mass change assessments (Zemp et al., 2019; Hugonnet et al., 2021). Figure 6 illustrates the compared time series of glacier change for the three global assessments including the glacier change product CDR. The Hugonnet et al., (2021) stands as the benchmark for long term trends of glacier change providing individual glacier elevation changes from the geodetic method for an almost complete observational sample (96% of the global glaciers). As a drawback, it is not able to capture the annual temporal variability of glacier mass changes and it covers only the present period (2000 to 2020). Contrarily, Zemp et al., (2019) is able to capture the annual variability of glacier mass changes at a regional level and, at the same time, go back in time. However, the long-term trend is biased mostly due to the scarce observational sample (9% of global glaciers).
The new algorithm to produce the glacier change product CDR is able to i) correct the bias of Zemp et al´s. (2019) long-term trend by integrating the glacier elevation changes of Hugonnet et al. (2021) in the processing chain. (ii) Keep the annual temporal variability of glacier mass changes back in time until 1976 at a global level, and (iii) considerably reduce the uncertainties of the annual values thanks to an improved observational sample (96% of global glaciers) and further improvements of the algorithm.
Figure 6: Global annual glacier mass change time series and sea level equivalent for the three existing global assessments: Zemp et al., (2019) in Blue, Hugonnet et al., (2021) in Yellow and the glacier change product CDR version WGMS-FOG-2023-09 in Purple.
3.2. Comparison to independent global glacier mass change assessments
Figure 7 compares the glacier change product CDR global mean against an independent assessment of global glacier mass changes using gravity based methods derived from the Gravity Recovery and Climate Experiment (GRACE) satellite (Wouters et al., 2019). The good agreement between the two independent assessment during their common time periods (2002 – 2021) further supports the validity of the glacier change CDR.
Figure 7: Global cumulative glacier mass changes from April 2002 to December 2021 from Wouters et al., (2019) (Yellow) and from April 2002 to March 2022 from the glacier mass-change product CDR version WGMS-FOG 2023-09 (Purple). For the sake of comparison, the annual values of the glacier change CDR are evenly distributed into months through the hydrological year and accumulated on a monthly basis to fit the monthly temporal resolution of GRACE estimates.
4. Compliance with user requirements
The related characteristics of the global gridded glacier mass change product of interest to the Global Climate Observing System (GCOS) requirements glacier ECV are shown in Table 1. Our product is a major step forward – the first globally gridded product – and mostly fulfills the GCOS requirements at the level of its input data. The final global product aggregates glacier-wide results at a spatial resolution of 0.5°, which fits the resolution of most other ECVs in the CDS but is lower than the GCOS requirements for glaciers. An improvement of the spatial resolution would require that all input data is compiled and made available as distributed products at about 25x25m resolution (similar to Hugonnet et al. 2021). However, this would require a major increase of the financial and infrastructure resources of the WGMS.
For more details on the updated GCOS requirements we refer the reader to Product User Guide and Specification document (PUGS; RD3). Relevant information linked to gaps and compliance with data requirement in past versions of the CDS glacier mass change product CDRs, accounted for in the new global gridded annual glacier mass change product CDR, is available in the Target Requirements and Gap Analysis document (TRGAD; RD4).
Table 1: Characteristics of the C3S globally distributed annual glacier mass change product from the latest version of the Fluctuations of Glaciers database (WGMS, 2023).
Data product characteristic | C3S globally distributed annual glacier mass changes ECV Product Requirements | |
Unit | Value | |
Horizontal resolution | Gridded (WGS-84) | 0.5° |
Vertical resolution | N/A | |
Temporal resolution | year | 1 |
timeliness | year | Updated on a yearly basis |
Required measurement uncertainty | Gt per grid point | between 0.02-0.2 |
Standards and | Zemp et al., (2019, 2020); WGMS (2023); Dussaillant et al., (in prep) | |
References
Cogley, J. G., Hock, R., Rasmussen, L. A., Arendt, A. A., Bauder, A., Braithwaite, R. J., et al. (2011). Glossary of Glacier Mass Balance and Related Terms. IHP-VII Technical Documents in Hydrilogy No. 86. IACS Contribution No. 2 UNESCO-IHP, Paris.
Dussaillant, I., Hugonnet, R., Huss, M., Berthier, E., Paul, f., Zemp M. (in preparation) An annual mass balance estimate for each of the world's glaciers based on observations.
Hugonnet, R., McNabb, R., Berthier, E., Menounos, B., Nuth, C., Girod, L., et al. (2021). Accelerated global glacier mass loss in the early twenty-first century. Nature 592, 726–731. doi: 10.1038/s41586-021-03436-z.
WGMS (2023). Fluctuations of Glaciers Database. doi: 10.5904/wgms-fog-2022-09.
Wouters, B., Gardner, A. S., and Moholdt, G. (2019). Global Glacier Mass Loss During the GRACE Satellite Mission (2002-2016). Front. Earth Sci. 7. doi: 10.3389/feart.2019.00096.
Zemp, M., Frey, H., Gärtner-Roer, I., Nussbaumer, S. U., Hoelzle, M., Paul, F., et al. (2015). Historically unprecedented global glacier decline in the early 21st century. Journal of Glaciology 61, 745–762. doi: 10.3189/2015JoG15J017.
Zemp, M., Huss, M., Eckert, N., Thibert, E., Paul, F., Nussbaumer, S. U., et al. (2020). Brief communication: Ad hoc estimation of glacier contributions to sea-level rise from the latest glaciological observations. The Cryosphere 14, 1043–1050. doi: 10.3929/ethz-b-000408094.
Zemp, M., Huss, M., Thibert, E., Eckert, N., McNabb, R., Huber, J., et al. (2019). Global glacier mass changes and their contributions to sea-level rise from 1961 to 2016. Nature 568, 382. doi: 10.1038/s41586-019-1071-0.
Zemp, M., Thibert, E., Huss, M., Stumm, D., Rolstad Denby, C., Nuth, C., et al. (2013). Reanalysing glacier mass balance measurement series. The Cryosphere 7, 1227–1245. doi: 10.5194/tc-7-1227-2013.
