Last modified on Sept 04, 2023 16:39

Contributors: B. Calmettes (CLS), N. Taburet (CLS), L. Carrea (University of reading), C.J. Merchant (University of reading)

Issued by: CLS / B Calmettes

Date: 30/04/2021

Ref: C3S_312b_Lot4_D2.LK4-v1.0_PQAR_LWL_v1.

Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC1

History of modifications

List of datasets covered by this document

Related documents

Acronyms

General definitions

Accuracy: The closeness between the measured value and the true quantity value.

Precision: Closeness between measured values obtained by replicate measurements on the same object under similar conditions

Bias: Estimate of a systematic error

Scope of the document

This document is the Product Quality Assessment Report (PQAR). This document is meant to summarize the results from the product assessment based on the Product Quality Assurance Document (D2.LK.3-v3.0) for the C3S Lake Water Level product. Although, it can be made public, it is primarily intended to be shared with the Evaluation and Quality Control (EQC) activity at C3S.

Executive summary

The C3S lake production system (C3S ECV LK) provides an operational service, generating lake surface water temperature and lake water level climate datasets for a wide variety of users within the climate change community. The present document covers the lake water level system.

This document presents the results of the quality assessments undertaken for the product including absolute assessment by evaluating three performance indicators: ( i ) dispersion, (ii) high frequency variations and (iii) mean time step between estimation. It also includes relative assessment comparing C3S datasets to independent altimetry products as well as in-situ measurements.

This document concerns the version3.1 of the C3S LWL products.

1. Product validation methodology

1.1. Validated products

The Water Level is the measure of the absolute height of the reflecting water surface beneath the satellite with respect to a vertical datum (geoid) and expressed in metres. The C3S lakes products comprise a long-term climate data record (CDR). The time series has been computed from multiple altimetry satellites since late 1992 to 2020 inclusive. The time periods used for each satellite/instrument are provided in Table 1 but may vary from one lake to the other, depending on the orbits of the satellites with respect to the location of the lake.

Table 1: Time periods for the satellite/instrument used to generate the lake product

A detailed description of the product generation is provided in the Algorithm Theoretical Basis Document (ATBD) [D1.LK.4-v3.0] with further information on the product given in the Product User Guide and Specifications (PUGS) [D3.LK.6-v3.0]

1.2. Validating datasets

A combination of in-situ and external altimetry-based products are used to assess the quality of the C3S lakes products. The list of datasets used is provided in Table 2.

Table 2: Datasets used in the assessment of the data product split by altimetry-based and in-situ data

1.3. Description of product validation methodology

The quality assessment of the Lake Water Level product includes the comparison of the dataset to external data (in-situ and altimetry-based) as well as tests to determine the long-term stability of the product at climate scale.

1.3.1. Absolute assessment

The validation exercise consists of validating the local changes of water level as measured by altimetry. The uncertainties or errors in the products are induced by two categories of errors: measurement errors and processing errors. Altimeters measure the distance between satellite and lake surface. However, numerous perturbations should be considered, and corrections need to be subtracted to take into account various physical phenomena. Some are already evaluated as the geoid, the ionospheric correction, wet and dry tropospheric correction and polar tides. For other phenomena, it is not possible to correct for these effects because the information is not currently available operationally at global scale such as wind effects or "oceanic" tides in large lakes, even though this may induce an uncertainty of a few centimetres.

Measurement errors may thus have several causes but the dominant source of measurement errors in inland waters is the land contamination of the footprint: in some configurations, nearby land may be as echogenic as water and interfere with the echo. In this case, the range measurement, hence the water level measurement, may be affected. The main challenge of the processing is thus to correctly identify the valid measurements.

The Lake Water Level products may contain altimeter data from multiple satellites tracks as well as different missions. Transects (intersections between satellite tracks and lakes) are on average longer on large lakes. Since the land contamination of the footprint is the major source of error in the measurements, transects on large lakes have both a higher number and a higher percentage of measurements that are not contaminated by this type of errors. The precision is thus better for large lakes.

Three performance indicators are chosen to assess the quality of lake products:

• Dispersion: This metric quantifies the dispersion of the individual successive measurements recorded by the altimeter when flying above the lake at a given time. It thus quantifies the precision of the estimated Lake Water Level data at each time step of the time series.
• High-frequency variations: standard deviation of the high frequency signal within each time series (computed thanks to a Lanczos high-pass filter with an arbitrary 1-month cut-off period). This indicator gives additional information on the lake water level precision for small lakes. Indeed, it primarily quantifies remaining errors due to the geoid model, as well as the shifts in the satellite orbits and the inter-mission bias.
• Mean time step: Average time between two valid measures. Since the estimation of the lake water level is based on multiple missions with different repetition cycles and different ground tracks, the time step per lake is not regular. Moreover, measurements may also be missing due to the poor quality of data that has been automatically removed during the process. This indicator provides information on the average frequency of data available per lake.

These performance indicators are estimated for each lake on two time periods: the full time series of ~30 years for most lakes and the last 10 years. These last indicators give the performance of recent quality of the products and provides insight on the future quality of the next version of the TCDR and CDR LWL products.
Since LWL products are derived from multiple missions, other interesting indicators involve the comparison of the performance between missions. Missing values per mission are calculated for current missions: Jason-3,Sentinel-3A and Sentinel_3B

1.3.2. Relative Assessment

External products using different data processing or acquisition are useful to assess the quality of the Lake Water Level products. Two types of datasets are considered: data generated by altimetry products and data obtained by in-situ measurements. These products use different datums, different dates, and for the altimetry products, different altimetry missions or standards/tracks.

The comparison is not straightforward. However, it provides information on the product's precision and accuracy. It must however be thoroughly analysed to understand if the differences are within the products' uncertainties or errors in one of the two products.

For each lake, we estimate the difference of the variation, the bias and the Pearson coefficient to estimate the linear correlation between Lake Water Level time series.

2. Validation results

2.1. Absolute assessment

Three performance indicators are chosen to assess the quality of the lake water level products: dispersion, high frequency variations and mean time step. These values were estimated for each lake on two time periods: the full time series of ~30 years for most lakes and the last 10 years. Annex A contains the values of those performance indicators for each lake on the two time periods. Performance indicators of the last 10 years are indicative of recent and future quality of lake products. Figure 1 provides an overview of the performance indicators for all lakes and three lake size categories:

• Small lakes: with surface less than 3000 km²
• Medium lakes: with surface within 3000 and 10000 km²
• Big lakes: with surface greater than 10000 km²

These general results will be commented more thoroughly in the subsections. However, they illustrate the general behaviour explained above: the precision decreases with the size of the lakes. The dispersion improves between the full period and the last 10 years only, thanks to the improvement of the sensors but also the orbits and other geophysical corrections. However, the high-frequency variations increase because more satellites and more tracks are used in the products (decreasing the mean time step), inducing inter-calibration issues as well as uncertainties related to the limitations of the geoid models.

Figure 1: Performance indicators for the overall period (~30years) in dark colours compared to the last 10 years (2010-2019 period) in light colour for three categories of lakes depending on their size (less than 3000km², between 3000 and 10000 km² and bigger than 10000km²). Dispersion and High-frequency variations are indicated in cm and Time step in days

These indicators are based on information from 166 lakes available in version 3 compared to 94 lakes in version 2. The changes are due to the characteristics of the different lakes which have different effects depending on the indicator.

The overall dispersion has increased compared to version 2 mainly for small lakes, which are generally more contaminated by land surroundings. However, the reasons of this increasing dispersion are different in each the lake. For some of them, outliers increase the median dispersion, as is the case of lake Mangbeto. In other cases, the dispersion evolves over time and decreases in recent missions, as in the case of lakes Hyargas and Kairakum, or even increases as is the case of lake Tchany (Figure 2)

Figure 2: Dispersion for Lakes (a) Mangbeto (b) Hyargas (c) Kairakum (d) Tchany

The high frequency has decreased because several new lakes are crossed by a single mission which avoids bias between tracks and missions. Finally, the median Time step increased, mainly in the small lakes as some of them are only overpassed by the Sentinel-3A or Sentinel-3B mission with a revisiting time of 27 days.

2.1.1. Along-track dispersion

The median transect dispersion per lake is less than 10 cm for medium and big lakes in line to the threshold in the product requirements. For small lakes, the median dispersion has increased to more than 16 cm due to the size of new small lakes (minimal lake area of 179 km2 in version 2 compare to 34 km2 in version 3). As indicated in the previous section, land contamination in the footprint being one of the main source of error in altimetry, there is a higher probability to have land contamination with small lakes, which increases the dispersion, whereas big lakes tend to provide results similar to what is expected on oceanic surfaces for altimetry.

Figure 3 shows the dispersion per lake according to size. Lake Kossou, a small lake of 1700 km2 (less than 3000 km2) has the greatest dispersion. This is due to the position of the track over the lake (Figure 4). In this case, only part of the transect is used to estimate the water level and, giving its shape, the signal may be highly contaminated by land surrounding.

Figure 3: Dispersion according to lake surface (surface in logarithmic axis). Red lines separating the lakes per size category (less than 3000km², between 3000 and 10000 km² and bigger than 10000km²)

Figure 4: S3A Transect over Kossou Lake

In addition to an individual analysis by lake over the entire period, it is also important to analyse the evolution of dispersion over the past few years. This information is useful for assessing the quality of the most recent measurements, which is expected to be better, thanks to the improvement of the sensors and the ground segments. Figure 5 shows the dispersion for each lake for the full period and for the last 10 years (2010-2019) ordered by lake size only for lakes with at least 20 years of temporal coverage. As expected, the dispersion decreases for almost all of them.

Figure 5: Dispersion per lake (sorted by size) during the full period compared to the last 10 years for lakes with at least 20 years of temporal coverage

In a general way, the shape of the lake and the position of the ground tracks have a significant impact on the quality of the estimation of the lake water level. If we analyse the Lagoa dos Patos, a lake with a surface area of 10000 km², just in the limit of medium/big size lakes, the dispersion has considerably evolved with the altimetric missions (Figure 6). During the period TOPEX/Poseidon (Figure 7a), only one near land track is available. With GFO (1998), a second track, with a better location but also near land, crosses the lake (Figure 7b). Thanks to ENVISAT (2002), several well-positioned ground tracks are available (Figure 7c). In 2008, Jason-2 data improved the quality of the estimated LWL product along the same ground tracks as TOPEX/Poseidon. Finally, Sentinel-3a allowed an increase in quantity and quality of the lake water level estimation (Figure 7d) with several tracks over the lake and a globally improved system.

Figure 6: Water level and dispersion time series for the Lagoa do patos

(a) TOPEX/Poseidon ground tracks	(b) TOPEX/Poseidon + GFO ground tracks
(c) TOPEX/Poseidon + GFO + ENVISAT ground tracks	(d) TOPEX/Poseidon + GFO + ENVISAT + Sentinel3-a ground tracks

Figure 7: Ground Tracks over passing the Lagoa do Patos. a) TOPEX/Poseidon in red, GFO in green, ENVISAT in Yellow and Sentinel-3A in blue)

2.1.2. High-frequency variations

The second indicator concerns the high frequency signal (Figure 8). theses variations are evaluated by taking the standard deviation of 1-month high-pass filtered LWL time series. They mainly contain "noise" due to measurement uncertainty as well as the geophysical signal of the high-frequency water level variations. This variation over the last ten-year period is higher than over the full period. This is mostly because there have been more satellites, thus more individual measurements (lower time step), over the last 10 years. More measurements with their specific precision and geoid errors yield to an increased amplitude of the high-frequency signal. Measurement uncertainty estimates using this indicator are less than 10 cm on average, which are also within accuracy requirements for lake product.

Figure 8: High frequency variation per lake (sorted by size) during the whole period compared to the last 10 years for lakes with at least 20 years of temporal coverage.

One of the examples of the increasing of the high frequency variation is lake Kariba. Since 2016, 6 ground tracks of Jason 3 and Sentinel-3A overpass the lake (Figure 9). Thanks to that, the median time step decreased from 12,2 days to 7,3 days but the time series during the last period is a lot noisier (Figure 10).

Figure 9: Ground Tracks over passing lake Kariba (TOPEX and Jason in red, Sentinel-3A in blue)

Figure 10: Water level and dispersion time series for Lake Kariba

2.1.3. Time Resolution

The median time step between two valid water level measurement depends strongly on the tracks per mission overpassing the lake. Figure 11 clearly shows that some lakes are over passed by a single track of a mission, with a time step close to the satellite revisit time: 27 days for lakes with a track from Sentinel 3 mission or 10 days for lakes with a track from the Jason family missions.
Due to a greater number of satellite tracks sampling the lakes during the two-altimeter era, the frequency of measurements is much higher. Nevertheless, it concerns mainly big lakes, with more probability to be monitored by multiple missions and multiple tracks are impacted. . The other factor impacting the median time step is also the percentage of edited or missing measurements. With the improvement of the altimetry products (sensors, geophysical correction, ground segment, orbit…etc), less measurements are identified as outliers and edited. The increasing on the number of missions in different orbits as Sentinel 3B, may not have an impact on the time resolution but it will increase the spatial resolution.

Figure 11: Median time step per lake (sorted by size) during the all period compare to the last 10 years for lakes with at least 20 years of temporal coverage.

Another interesting indicator is the percentage of missing values. This value represents the number of lake water levels that could not be estimated for different reasons: quality of the signal, shift of the ground trajectory, fast change in the level that activates the editing of the estimate. These percentage values were estimated for the current missions: Jason-3, Sentinel-3A and Sentinel-3B, for all the lakes and for the three categories of lakes based on the size as defined in section 2.1. The missing values of Sentinel-3A are lower than missing values of Jason-3 (Table 3). The diagnostic of Sentinel -3B is quite different from the other active missions. Data from this mission have only been used since 2019 to measure water levels and the historical data and spatial coverage to provide a true diagnosis yet. The high number of eliminated measurements in big lakes is due to the threshold defining the maximal allowed variation. This threshold, specified for each lake and based on historical behavior deserved to be reviewed.

Table 3: percentage of missing values depending on the mission and the size of the lake

2.2. Relative assessment

2.2.1. Altimetry products

In the context of the time series comparison, for the altimetry products, monthly time series of LWL products from external sources (G-REALM, DAHITI) have been selected. The values are compared during the period 2010-2020 where all the processing and satellite systems are more stable.

2.2.1.1. Dahiti products

The Person coefficient give information about the correlation between timeseries, a value near to 1 indicating a very good correlation. The Figure 12 shows the Person coefficients between monthly timeseries from C3S and Dahiti. The lowest Pearson coefficient value is in Lake Van (0.44). Analysing timeseries for this lake (Figure 13),there is a shift between in C3S and Dahiti data values since end 2019, with C3S values being lower with a gradually decreasing. This decreasing in the water level is also confirmed by external observations indicating that this lake has experienced significative water loss due to evaporation in recent years (https://www.aa.com.tr/en/pg/photo-gallery/water-level-in-lake-van-dropped-due-to-climate-change/0).

Figure 12: Pearson coefficient between monthly time series from C3S and Dahiti datasets

Figure 13: Monthly Water level variation for Lake Van (red: C3S, blue: Dahiti) and difference between time series.

Similar pattern is observed for lakes Sevan and Uvs where after a good correlation during the period 2013 – 2019, the timeseries of the products started to diverge (Figure 14 and Figure 15 respectively).

Figure 14: Monthly Water level variation for Lake Sevan (red: C3S, blue: Dahiti) and difference between time series.

Figure 15: Monthly Water level variation for Lake Uvs (red: C3S, blue: Dahiti) and difference between time series.

The value of the RMS can also provide important information on the distance of time series. This value was evaluated for each lake on a monthly basis (Figure 16). For all the lakes, this value is less than 1m. The maximum value concerns lake Toktogul, where the high RMS value is driven by a period of missing data in the Dathiti time series, but both time series have the same seasonality (Figure 17).

Figure 16: RMS values C3S - Dahiti

Figure 17: Lake Toktogul – Comparaison to Dahiti

Analysing the comparison figures for all the lakes, another effect is identified. For some lakes, a jump in the unbiased difference was detected as it's the case for Lake Tanganica (Figure 18) most probable reason for these discrepancies is a different estimation of the intermission biases. More analyses are needed to identify which of the products contain an error and to correct for this anomaly.

Figure 18: Lake Tanganyika - Comparison to Dahiti products

2.2.1.2. G-REALM products

Similar to Dahiti products, there is a very good correlation with the lakes in G-REALM products (Figure 19). Indeed, most of the Pearson coefficients are above 0.8.

Figure 19: Pearson coefficient between monthly time series from C3S and G-REALM datasets

The lowest value of the Pearson coefficient is found in Lake Tchany (0.16). The Lake Tchany time series is very noisy (Figure 20) and this is probably due to its shape and the position of the tracks over the lake (Figure 21). Moreover, after the end of the Envisat mission, only a Jason 3 track can be used to estimate the water level, and its passage at the edge of the lake increases the probability of land contamination.

Figure 20: Lake Tchany - Comparison to G-REALM products

Figure 21: Ground Tracks over passing lake Tchany (TOPEX and Jason in red, ENVISAT in Yellow)

The second indicator to compare the datasets is the RMS value between time series. For the comparison to G-REALM product, Figure 22 shows these values for all common lakes on a monthly basis. Most of the lakes have an RMS value of less than 1m except for Lake Toktogul.

Figure 22: RMS values C3S - G-REALM

Concerning Toktogul lake, the location of the track (Sentinel 3-B) is very interesting, just at the extremity of the lake (Figure 23). This implies that the probability of editing the water level value due to land contamination is very high and, therefore, there are some periods without reliable data. Figure 24 shows timeseries comparing C3S data to the other altimetry-based datasets: G-REALM and Dahiti (ending in 2016), showing a difficult estimation.

Figure 23: Ground Tracks over passing lake Toktogul (Sentinel 3B)

Figure 24: Lake Toktogul - Comparison to G-REALM and Dahiti products

For the other lakes with low Pearson coefficient, in some cases is due to outliers in the timeseries, either from G-REAL or C3S (Figure 25, Figure 26, Figure 27 and Figure 28).

Figure 25: Lake Har - Comparison to G-REALM products

Figure 26: Lake Sasykol - Comparison to G-REALM products

Figure 27: Lake Hovsgol - Comparison to G-REALM products

Figure 28: Lake Hovsgol - Comparison to G-REALM products

2.2.2. In-situ products

2.2.2.1. Hydrolare

Thanks to the collaboration with the International Data Centre on Hydrology of Lakes and Reservoirs, Hydrolare, the information on four lakes was provided in a monthly time step. Three indicators were estimated: Bias, RMS and Pearson coefficient (Table 4). For all the lakes the Pearson coefficient is higher than 0.9. In some cases, the bias is high (up to 2.3m) and suggest that, even if the precision is good, the accuracy of the LWL products should be considered carefully by users. Annex B contains the figures corresponding to the time series of the variation for each of the lakes.

Table 4: Hydroweb – Hydrolare Indicators

2.2.2.2. Hidricos Argentina

The information concerning the historical variation on the Water Lever for several insitu stations in Argentina was obtained online from the Base de datos Hidrologica integrada (BDHI): http://bdhi.hidricosargentina.gob.ar. The RMS values and Pearson coefficients for five lakes are indicated in Table 5.

Table 5: Hydroweb – Hididricos Argentina Indicators

Only San Martin lake has a Pearson coefficient below 0.8. This is due to a temporal difference in the time series during 2018 that may be also generated by a problem in the in-situ measurements (Figure 29).

Figure 29: Lake San Martin - Comparison to Hidricos Argentina in situ dataset

Annex C contains the figures corresponding to the time series of the variation for each of the lakes.

2.2.2.3. US Army Corps of Engineer

The third source of in situ data is the US Army Corps of Engineers. The data for the Great Lakes in the USA is available online. Table 6 contains the indicators for the Great Lakes and the figures for each lake are in Annex D. The results show an excellent agreement between the present product and this in situ dataset, with RMS below 4cm and correlations above 0.98 even if some outliers are detectected mainly in 2014 and 2015.

Table 6: C3S – US Army Corps of Engeneer Indicators

2.2.2.4. US Geological Survey

The USGS provides information on water resources data collected mainly in US. The USGS investigates the occurrence, quantity, quality, distribution, and movement of surface and underground waters and disseminates the data to the public, State and local governments, public and private utilities, and other Federal agencies involved with managing our water resources. The information concerning the monthly lake water level evolution for lakes Ontarin, De Bois and Erie was obtained on-line. The RMS values and Pearson coefficient are in indicated in Table 7.

Table 7: C3S –USGS Indicators

The correlation between time series for all lakes is very high. It is confirmed by Pearson coefficients close to 1. For Lake De Bois, only the level variation is available and the bias for this lake can't be provided. Annex E include the comparison of variation of those lakes between C3S dataset and USGS dataset.

2.2.2.5. Water office of Canada

The in-situ data for Canadian lakes is freely available at the Water Office of Canada. The time series of thirteen lakes monitored in the C3S project were compared to this data and the Pearson coefficient, indicating the correlation between time series is most of the time higher that 0.8 (Figure 30).

Figure 30: Pearson coefficient between monthly time series from C3S and Water Office datasets

Concerning lake Winnipeg, the low value of the Pearson coefficient is due to some outliers in the in-situ measurements (Figure 31).

Figure 31: Lake Winnipeg - Comparison to Water Office of Canada in situ dataset

2.2.2.6. Federal Office for Environment (FOEN)

The FOEN defines environmental monitoring programs and maintains various measurement network. It operates and coordinates several water related monitoring networks. Moreover, it monitors the level of water in Switzerland rivers and lakes. Currently, two Switzerland lakes are monitored in the C3S project: Bodensee and Leman lakes. Table 8 contains the values of the RMS, bias and Pearson Coefficients comparing the timeseries from both datasets.

Table 8: C3S –FOEN Indicators

The Pearson coefficient for both lakes is close to one, indicating a very good correlation. The figures with the comparison of the timeseries are in Annex G.

3. Application(s) specific assessments

Currently, no application(s) specific assessments have been undertaken for the January 2021 version of the C3S lake water level dataset.

4. Compliance with user requirements

The requirements for the C3S Lake water levels are described in the Target Requirements and Gap Analysis document (D1.LK.1-v1.0).

Acknowledgements

Thanks to Prof. Valery Vuglinskiy for providing us with in-situ data from Hydrolare.
Thanks to C. Schwatke and the DAHITI team for sharing their database and reviewing the results.

References

Bercher, N. (2008). Précision de l'altimétrie satellitaire radar sur les cours d'eau: Développement d'une méthode standard de quantification de la qualité des produits alti-hydrologiques et applications (Doctoral dissertation).

Biancamaria et al., 2017, Validation of Jason-3 tracking modes over French rivers, Rem. Sens. Env., submitted

Chelton, D. B., Esbensen, S. K., Schlax, M. G., Thum, N., Freilich, M. H., Wentz, F. J., ... & Schopf, P. S. (2001). Observations of coupling between surface wind stress and sea surface temperature in the eastern tropical Pacific. Journal of Climate, 14(7), 1479-1498.

Crétaux, J. F., Abarca-del-Río, R., Berge-Nguyen, M., Arsen, A., Drolon, V., Clos, G., & Maisongrande, P. (2016). Lake volume monitoring from space. Surveys in Geophysics, 37(2), 269-305.

Jekeli, C., & Dumrongchai, P. (2003). On monitoring a vertical datum with satellite altimetry and water-level gauge data on large lakes. Journal of Geodesy, 77(7-8), 447-453.

Schwatke, C., Dettmering, D., Bosch, W., and Seitz, F.: DAHITI – an innovative approach for estimating water level time series over inland waters using multi-mission satellite altimetry, Hydrol. Earth Syst. Sci., 19, 4345-4364, https://doi.org/10.5194/hess-19-4345-2015, 2015

Annex A. Performance indicators

Annex B. Comparison to Hydrolare Data

Annex C. Comparison to Hidricos Argentina

Annex D. US Army Corps of Engineer

Annex E. US Geological Survey

Annex F. Water Office of Canada

Annex G. Federal Office for the Environment