Last modified on Jun 24, 2024 12:00

Contributors: B. Calmettes (CLS), G. Calassou (CLS), N. Taburet (CLS)

Issued by: CLS / B. Calmettes, G. Calassou

Date: 10/04/2024

Ref: C3S2_312a_Lot4.WP2-FDDP-LK-v2_202312_LWL_PQAR-v5_i1.1

Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1

History of modifications

List of datasets covered by this document

Related documents 

Acronyms

General definitions 

Accuracy: The closeness between the measured value and the true quantity value.

Precision: Closeness between measured values obtained by replicate measurements on the same object under similar conditions.

Bias: Estimate of a systematic error.

Lake Water Level: Measure of the absolute height of the reflecting lake water surface beneath the satellite with respect to a vertical datum (geoid) and expressed in metres.

Lake Water Level – Single track: corresponds to the Lake Water Level measure for lakes being observed by a single mission/track and estimated with a different algorithm.

Dispersion: Describes the degree of variation of successive measurements. This performance indicator provides information about the precision of the estimated data.

High-frequency variations: Variation of the high frequency signal because of errors due to models or bias.

Mean time step: Average time between two valid measures.

Scope of the document

This document is the Product Quality Assessment Report (PQAR) for the Copernicus Climate Change Service (C3S) Lake Water Level (LWL) and Lake Water Level – Single-track Lakes (LWL-S) products. It presents results of the quality assessment for the provided datasets according to the validation methods and strategies described in the Product Quality Assurance Document [D4].

Executive summary

The C3S Lake production system (C3S ECV LK) provides an operational service generating lake water level climate datasets. The dataset collections include one for medium to large lakes that considers the geoid variation along the track and the estimation of bias between tracks (LWL), and one for lakes observed only by a single track of an altimetry mission satellite (LWL-S) and usually with small surface area. These collections are made available for a wide variety of users within the climate change community. 

The quality assessment analysis for the lake water level products consists of two distinct parts: i) assessing the absolute error with the validation of the data, and (ii) assessing the relative error estimate by comparing generated products with external data. Quantifying absolute error was performed by analysing the error generated by the instruments and processing over time. The C3S lake water level product is based on measurements from several altimetry missions, with technology that has been developed and improved in consecutive missions (going from standard altimeters as Low Resolution Mode (LRM) onboard of Jason-3 to Synthetic Aperture Radar (SAR) onboard Sentinel-6A). Estimating relative error is achieved by comparing the generated products with external data from either i) other altimetry-based products or (ii) products derived from in-situ measurements.

This document presents the results of the quality assessments undertaken for both products. The product validation and methodology are described in Section 1, including i) a description of the lake water level product being analysed and (ii) the description of the different external products used for its validation. Section 2 details the validation results, both absolute and relative assessment. Section 3 concerns applications specific assessments, noting that there were none available for inclusion in this section at the time of publication of this document. Section 4 contains the summary of compliance of the generated lake water level datasets with respect to the user requirements.

1. Product validation methodology

1.1. Validated products

The Water Level is the measure of the absolute height of the reflecting water surface beneath the satellite with respect to a vertical datum (geoid) and expressed in metres. The C3S lakes products comprise a long-term climate data record (CDR). The timeseries has been computed from multiple altimetry satellites extending from late 1992 to 2023 inclusive. While the LWL dataset is computed both from past and current missions (Table 1), the LWL-S dataset is computed only from missions that are currently in operation. The time periods used for each satellite/instrument are provided in Table 1, noting that this may vary from one lake to the other, depending on the orbits of the satellites with respect to the location of the lake.

Table 1: Time periods for the satellite/instrument used to generate the lake product.

A detailed description of how the products are generated is provided in the Algorithm Theoretical Basis Document (ATBD) [D3], with further information on the products given in the Product User Guide and Specifications (PUGS) [D5]. 

1.1.1. LWL Dataset

The Lake Water Level is provided by regions (Table 2). The aim is to quickly identify the geographical area where a given lake is located. However, the lakes are not uniformly distributed in these regions. Figure 1 to Figure 11 show the position of the lakes in each region.

Table 2: Regions defined for the C3S LWL product.

Figure 1: Lakes in the C3S – LWL dataset v5.0 located in northern North America (34 lakes).

 
Figure 4: Lakes in the C3S – LWL dataset v5.0 located in Northern Europe (17 lakes).

Figure 5: Lakes in the C3S – LWL dataset v5.0 located in Southern Europe (6 lakes).

Figure 6: Lakes in the C3S – LWL dataset v5.0 located in Northern Africa (20 lakes). 

Figure 7: Lakes in the C3S – LWL dataset v5.0 located in Southern Africa (21 lakes).

Figure 8: Lakes in the C3S – LWL dataset v5.0 located in Northern Asia (15 lakes).

Figure 11: Lakes in the C3S LWL dataset v5.0 located in Oceania (3 lakes). 

1.1.2. LWL-S dataset 

Similarly to LWL products, LWL-S products are also provided by region (definition of the regions is the same as for the LWL dataset, see Table 2). Figure 12 to Figure 22 illustrate the location of the LWL-S lakes in each region.  

Figure 12: Lakes in the C3S – LWL-S dataset v1.0 located in northern North America (3014 lakes).

 
Figure 13: Lakes in the C3S – LWL-S dataset v1.0 located in southern North America (1221 lakes).

Figure 14: Lakes in the C3S – LWL-S dataset v1.0 located in South America (504 lakes).

Figure 15: Lakes in the C3S – LWL-S dataset v1.0 located in Northern Europe (904 lakes).

Figure 16: Lakes in the C3S – LWL-S dataset v1.0 located in Southern Europe (186 lakes).

Figure 17: Lakes in the C3S – LWL-S dataset v1.0 located in Northen Africa (110 lakes).

Figure 18: Lakes in the C3S – LWL-S dataset v1.0 located in Southern Africa (129 lakes).

Figure 19: Lakes in the C3S – LWL-S dataset v1.0 located in Northern Asia (1300 lakes).

1.2. Validating datasets

A combination of in-situ and independent altimetry-based products are used to assess the quality of the C3S lakes products. The list of datasets used is provided in Table 3.

Table 3. Datasets used in the assessment of the LWL data product split by altimetry-based and in-situ data.

1.3. Description of product validation methodology

The quality assessment of the LWL and LWL-S products involved the comparison of the dataset to external data (in-situ and altimetry-based), as well as tests to determine the long-term stability of the product at a climate scale. An overview of the methodology is presented here. However, for a more comprehensive description, please see the associated Product Quality Assurance Document (PQAD; [D4]).

1.3.1. Absolute error assessment

Altimeters measure the distance between the satellite and lake surface, with numerous processing steps needed to derive accurate estimates of this distance from the radar signal. The uncertainties or errors in the LWL and LWL-S products are induced by two categories of errors: measurement errors and processing errors.

Measurement errors may have several causes. Numerous influences on the radar signal should be considered, and corrections need to be applied to take into account various physical phenomena (for further information see the ATBD [D3]). Some are already evaluated as the geoid, the ionospheric correction, wet and dry tropospheric corrections and earth and polar tides. However, for other phenomena, it is not possible to correct for these effects because the information (the corrections) is not currently available operationally at global scale (such as for wind effects or “oceanic” tides in large lakes), even though this may induce an uncertainty of a few centimetres. Processing errors are linked to the estimation of parameter files containing, for each lake, intermission and inter-track biases and the estimated maximum variation in the water level.

For inland waters, the dominant source of measurement errors is land contamination of the footprint in some configurations. Nearby land may be as echogenic as water and interfere with the radar echo. In this case, the range measurement, hence the water level measurement, may be affected. The main challenge of the deriving viable LWL and LWL-S estimates is therefore to correctly identify the valid measurements.

Additionally, the Lake Water Level products may contain altimeter data from multiple satellites tracks as well as different missions. Transects (intersections between satellite tracks and lakes) are on average longer on large lakes. Since the land contamination of the footprint is the major source of error in the measurements, transects on large lakes have both a higher number and a higher percentage of measurements that are not contaminated by this type of errors. The precision is thus better for large lakes.  For lakes assessment, the precision of the measurements provides reliable information on lake water variation, whereas accuracy refers to a relative measurement, based on the datum used as reference.

Three performance indicators have been chosen to assess the quality of lake products in terms of their absolute error:

• Dispersion: This metric quantifies the dispersion of the individual successive measurements recorded by the altimeter when flying above the lake at a given time. It thus quantifies the precision of the estimated LWL data at each time step of the timeseries.
• High-frequency variations: standard deviation of the high frequency signal within each timeseries (computed thanks to a Lanczos high-pass filter (Lanczos, 1988) with an arbitrary 1-month cut-off period). This indicator gives additional information on the lake water level precision for small lakes. For consistency reasons, it is estimated for lakes of all sizes. Indeed, it primarily quantifies remaining errors due to the geoid model, as well as the shifts in the satellite orbits and the inter-mission bias.
• Mean time step: Average time between two valid measures. Since the estimation of the lake water level is based on multiple missions with different repetition cycles and different ground tracks, the time step per lake is not regular. Moreover, measurements may also be missing due to the poor quality of data that has been automatically removed during the process. This indicator provides information on the average frequency of data available per lake.

The performance indicators were estimated for LWL product based on three categories of lake size:

• Small lakes: with surface areas of less than 3000 km²
• Medium lakes: with surface areas between 3000 and 10000 km²
• Large lakes: with surface areas greater than 10000 km²

For LWL-S product, the performance indicators were evaluated based on three categories of transect length:

• Small transect: transects with a length shorter than 1 km.
• Medium transect: transects with a length between 1 and 5 km.
• Large transect: transects with a length greater than 5 km.

Additionally, for the LWL dataset, the performance indicators were estimated for two time periods: the full timeseries of ~30 years for most lakes and the last 10 years. These indicators based on the last 10 years give the performance of recent quality of the products and provides insight into the future quality of subsequent versions of the Thematic CDR (TCDR) and CDR LWL products.

Since LWL products are derived from multiple missions, other interesting indicators involve the comparison of the performance between missions. Missing values per mission are calculated for current missions: Sentinel-3A, Sentinel-3B and Sentinel-6A and one past mission: Jason-3.

1.3.2. Relative error assessment

External products using different data processing or acquisitions are useful to assess the quality of the LWL products. Two types of datasets are considered: data generated by altimetry products, and data obtained by in-situ measurements. These products use different datums, different dates, and, for the altimetry products, different altimetry missions or standards/tracks. Thus, the comparison is not straightforward. However, it can provide information on the product's precision which are examined through this relative error assessment. It must however be thoroughly analysed to understand if the differences are within the products' uncertainties or errors in one of the two products.

For each lake, three metrics are evaluated:

• Pearson coefficient: this coefficient provides information about the correlation between two timeseries. A value near to 1 indicates that a very good correlation exists between two timeseries.
• Unbiased Root Mean Square Error (URMSE): this measure is based on the difference of the times series after removing the bias component.
• Bias: As described in the previous paragraph, the reference for the water level estimation changes between the different datasets and there is often a bias between timeseries. This bias is only evaluated for the LWL product.

2. Validation results

2.1. Absolute error assessment

2.1.1. Lake Water Level (LWL) Dataset

The three performance indicators described above (see Section 1.3.1) were estimated for each lake (i.e., 251 lakes available in version 5) for two time periods: the full timeseries of ~30 years for more than 70 lakes, and the last 10 years. Annex A contains the values of those performance indicators for each lake for the two time periods. Performance indicators for the last 10 years are indicative of the recent and future quality. Figure 23 shows the proportion of measured lakes by their size in the LWL version 5 product. Meanwhile, Figure 24 provides an overview of the performance indicators for all lakes and three lake size categories (section 1.3.1).

Figure 23: Distribution of lakes by size.

These general results are described more thoroughly in the subsections below. However, they illustrate the general behaviour noted below, i.e., that the precision decreases with the size of the lakes. The dispersion decreases between the full period and the last 10 years only. This is most likely due to the improvement of the sensors, but also the increase in the precision of orbits measurements thanks to DORIS system (Doppler Orbitography and Radiopositioning by Satellite) and other geophysical corrections. Additionally, a higher number of samples, for the full period, could also have an impact in the dispersion estimation. However, the high-frequency variations increase because more satellites and more tracks are used in the products (decreasing the mean time-step), which induces inter-calibration issues as well as uncertainties related to the limitations of the geoid models. 

Figure 24: Performance indicators for the overall period (1992-2022). 156 lakes have a temporal coverage of more than 10 years) in dark colours compared to the last 10 years (2014-2023 period) in light colours for three categories of lakes depending on their size (arranged left to right per indicator group as small lakes: less than 3000km², medium lakes: between 3000 and 10000 km², and large lakes: larger than 10000 km²).

The overall dispersion in version 5 (251 lakes) is similar to version 4 (229 lakes). The dispersion decreases over the latter years in the series, because more recent missions are performing better than previous missions. Nevertheless, the dispersion of the lakes can be very variable. Depending on the location of the track over the lake, mainly near the shore, the backscatter signal may be affected by land contamination such as the proximity of other water bodies or any reflecting surface. For some of them, outliers increase the median dispersion, as is the case of Lake Mangbeto (Togo) which is shown in Figure 25. As mentioned above, in most of the cases, the dispersion changes over time and decreases in recent missions, as shown in Figure 26 for Lake Hyargas (Mongolia).

Compared to previous versions, the high frequency variation in LWL version 5 has decreased because several new lakes are crossed by a single mission, which avoids bias between tracks and missions. Finally, the median Time step increased, mainly in the small lakes, as some of them are only observed by the Sentinel-3A or Sentinel-3B mission with a revisiting time of 27 days.

Figure 25: Dispersion of the estimated LWL data over time for Lle Mangbeto (Togo).

Figure 26: Dispersion of the estimated LWL data over time for Lake Hyargas (Mongolia).

2.1.1.1. Along-track dispersion

The median transect dispersion per lake is less than 10 cm for medium and large lakes, in line with the threshold in the product requirements (3.5 cm and 6.9 cm, respectively). For small lakes, the median dispersion is 11.34 cm for the overall period but decreases over the recent 10-year period. Several of these small lakes are monitored by recent missions, such as Sentinel-3A and Sentinel-3B. These missions feature improved along-track resolution (approximately 300 m) in SAR mode which facilitates the measurement of small lakes. As indicated in the previous section, land contamination in the footprint is one of the main sources of error in altimetry over inland waters. There is a higher probability to have land contamination with small lakes, which increases the dispersion, whereas large lakes tend to provide similar results for altimetry to what is expected for oceanic surfaces.

Figure 27 shows the dispersion per size of the lake for those lakes with at least 20 years of data coverage. It shows how the median dispersion increases with the size of the lakes. Particularly for small lakes which are more impacted by land surroundings, the number of lakes with high dispersion increases. The performance of the mission has a great impact in this dispersion. For example, Lake Lagdo (Cameroon), a small lake of 586 km², has a mean dispersion of 25cm, with a great variability over time, depending on the mission monitoring the lake (Figure 28). This lake was initially monitored by Envisat (2002-2012) and is currently being monitored by Sentinel-3A, starting in 2016. The Cryosat data have made it possible to fill some of the data gaps in this lake between 2012 and 2016.

Figure 27: Lake Water Level dispersion in relation to lake size, for those lakes with over 20 years of data coverage (130 lakes).

 

Figure 28: Temporal development of dispersion of LWL estimates for Lake Lagdo (Cameroon), a small lake of 586 km² with mean dispersion of 25 cm.

In addition to an individual analysis by lake over the entire period, it is also important to analyse the changes in the dispersion over the past few years. This information is useful for assessing the quality of the most recent measurements, which is expected to be better, thanks to the improvement of the sensors and the ground segments. Figure 29 shows the mean dispersion for all lakes over the full period (1992-2023) and over the last 10-year (2014-2023) for lakes with at least 20 years of temporal coverage. As expected, the median of the dispersion decreases from 7.25 cm for the complete period to 6 cm for the last 10-year period. There is also a decrease of the number of outliers and their values. 

Figure 29: Boxplot showing the dispersion per lake for the full period (1992-2023) and the last 10 years (2014-2023) for lakes with at least 20 years of data coverage (130 lakes).

In general, the shape of the lake and the position of the ground tracks have a significant impact on the quality of the lake water level estimation. If we analyse the Lagoa dos Patos (Brazil), a lake with a surface area of 10000 km², just at the threshold between medium to large size lakes, the dispersion has changed considerably with the altimetric missions (Figure 30). During the period covered by TOPography EXperiment (TOPEX)/Poseidon (Figure 31a), only one near-land track is available. With Geosat-Follow-On (GFO) (launched in 1998), a second track, with a better location but also near land, crosses the lake (Figure 31b). Thanks to ENVISAT (launched in 2002), several well-positioned ground tracks became available (Figure 31c). Then in 2008, Jason-2 data improved the quality of the estimated LWL product along the same ground tracks as TOPEX/Poseidon. Finally, Sentinel-3A enabled an increase in the quantity and quality of lake water level estimation (Figure 31d) with several tracks over the lake and a globally improved system.

Figure 30: Change over time of the dispersion for the Lagoa dos Patos in Brazil (surface area of 10,000 km², a medium-large size lake). The dispersion has changed considerably with the advent of new altimetric missions to the measuring constellation (see also Figure 31).

Figure 31: Ground Tracks over passing the Lagoa dos Patos in Brazil. TOPEX/Poseidon in red, GFO in green, ENVISAT in Yellow and Sentinel-3A in blue).

2.1.1.2. High-frequency variations

The second indicator concerns the high frequency signal variations. They mainly contain "noise" due to measurement uncertainty as well as the geophysical signal of the high-frequency water level variations. The variation over the recent 10-year period (2014-2023) is slightly higher than over the full period for lakes with at least 20 years of temporal coverage (5.94 cm and 6.05 cm respectively). This is mostly because there have been more satellites, thus more individual measurements (lower time step), over the last 10 years. More measurements with their specific precision and geoid errors yield an increase in high-frequency signal amplitude. Measurement uncertainty estimates using this indicator are less than 10 cm on average, which are also within accuracy requirements for the lake product (see Section 4).

Figure 32: Boxplot showing the high frequency variation for the full period (1992-2023) and the last 10 years (2014-2023) for lakes with at least 20 years of data coverage (130 lakes).

One of the examples of increasing high frequency variation over 1992-2022 is Lake Kariba located along the border between Zambia and Zimbabwe. Since 2016, six ground tracks of Jason-3 and Sentinel-3A overpass the lake (Figure 33). Thanks to that, the median time step decreased from nine days to four days, although the timeseries during the last period is noisier (Figure 34). 

Figure 33: Ground Tracks over passing Lake Kariba between Zambia and Zimbabwe (TOPEX and Jason in red, Sentinel-3A in blue) since 2016.

 

Figure 34: Dispersion timeseries for Lake Kariba located along the border between Zambia and Zimbabwe.

2.1.1.3. Time Resolution

The median time step between two valid water level measurements strongly depends on the tracks per mission observing the lake. Figure 35. shows the boxplot with the time step distribution for the overall period and the last 10-year period for lakes with at least 20 years of temporal coverage. The time step clearly decreases in the last 10-year period but the number of outliers increases. In fact, some new lakes are observed only by a single recent mission as Sentinel 3-A/B with a revisit period of 27 days.

Figure 35: Boxplot showing the distribution of the time step for the full period (1992-2023) and the last 10 years (2014-2023) for lakes with at least 20 years of data coverage (130 lakes).

Another interesting indicator is the percentage of missing values. This value represents the number of lake water level estimates that could not be derived from their associated altimetry echo for different reasons: quality of the signal, shift of the ground trajectory, fast change in the level that activates the editing of the estimate. These percentage values were estimated for the current missions: Sentinel-3A, Sentinel-3B and Sentinel-6A and one past mission: Jason 3 with the same orbit as Sentinel-6A, for all lakes (251 in version 5) ,and for the three categories of lake size as defined in section 2.1 (Table 4). 

Table 4: Percentage of missing values depending on the mission and the size of the lake.

* Currently, only one large lake, Lake Bagre, in Burkina Faso is monitored using data from Sentinel-3B

2.1.2. Lake Water Level – Single-track (LWL-S) Dataset

For the LWL-S products, three performance indicators described in Section 1.3.1 were computed for each lake (i.e., 8537 lakes available in C3S LWL-S version 1.0 dataset) for the entire measurement period of each used mission. As described in section 1.3.1, the performance was evaluated by transect length. The distribution of the transect lengths is illustrated in Figure 36. Most of the monitored lakes have a transect length between 1 and 5 km (71 %) (see Figure 37). Figure 38 illustrates the transect size in relation to the lake area and the two parameters appear to be correlated. However, for some cases, a small transect doesn't imply that the surface of the monitored lake is small, because measurements could be taken at the lake's edges.

Figure 36: Distribution of the transect length in the LWL-S dataset.

Figure 37: Proportion of transects by length category.

Figure 38: Area of lakes in LWL-S dataset in relation to the transect length.

 

Figure 39: Boxplot of the mean dispersion by transect by lake according to the classification of the transect length (small, medium and large transect).

2.1.2.1. Along track dispersion and completeness of timeseries

The median transect length dispersion per transect equals approximately 10 cm for medium and small transect categories (10.0 cm and 9.3 cm respectively). For large transects, the median transect length dispersion is equal to 20.4 cm. 

These results may seem counter intuitive compared to observations on LWL products. However, for a specific date, the chosen water level value within a timeseries corresponds to the median of the water level values distribution along a transect (see the Algorithm Theoretical Basis Document (ATBD) [D3]). Despite the high dispersion of measurements within a large transect, the selected value for the associated timeseries remains consistent with those observed at previous time steps. In contrast, in a small Sentinel-3 transect, which may consist of only four measurements (with a 300 m interval between each measurement), more than half of these measurements, at any given date, may be influenced by the lake shore. Consequently, the selected water level will deviate significantly from the expected water level. It will therefore be detected as an anomaly by the editing process applied to the timeseries and will therefore be rejected. Thus, the only values retained in lake timeseries with small transects are those with a low dispersion.

That is why it is crucial to analyze dispersion results considering the completeness score of timeseries acquired for each monitored lake. Despite small transects yielding better dispersion results, the completeness of the timeseries is influenced by their size.

Figure 40:Boxplot illustrating dispersion of the completeness of lake timeseries according to the category of their associated transect.

Figure 40 illustrates the completeness of the timeseries for each transect category. As expected, large transects exhibit higher completeness than small ones. The completeness of each transect category is equal to 80.2 %, 68.6 % and 63.8 % for large, medium and small transects, respectively.

2.1.2.2. High frequency variations

High frequency variations are calculated as the standard deviation of residuals, which are obtained from the differences between the water level timeseries and the same timeseries re-built as the sum of its trend and seasonal components. The trend and the seasonal components are computed using a Seasonal Trend decomposition by LOcally Estimated Scatterplot Smoothing (LOESS) (STL) (Cleveland et al., 1990).

Similar to LWL products, this way of calculating the high frequency variations allows for the analysis of both the noise caused by the measurement uncertainty and the geophysical signal of the high frequency variations of lake water level. Figure 41 illustrates the high frequency variation for each category of transect. The median of high frequency variations for small, medium and large transects equals to 17.5 cm, 18.1 cm and 21.4 cm respectively. The high frequency variations for large transects is higher than the other categories due to a higher dispersion along the transect (see Figure 39).

Figure 41: High frequency dispersion for each transect category.

2.2. Relative error assessment

2.2.1. Lake Water Level (LWL) Dataset

2.2.1.1. Altimetry products

Timeseries from C3S LWL v5.0 dataset were compared with products from two datasets based on altimetry datasets: G-REALM and DAHITI (see Table 3). The three metrics, described in section 1.3.2, were analysed for each of the eleven regions described in Table 2. These metrics were estimated on a daily basis, considering only the measurements available in both (C3S and G-REALM or DAHITI) datasets on the same day.

It is important to note that the number of lakes monitored by the different datasets (from C3S and external sources) is not the same. As such, there are differences between the number of lakes monitored by C3S in the different regions and the number of lakes compared to external datasets. The results of this analysis focus on common lakes.

2.2.1.1.1. Pearson coefficient

Figure 42 and Figure 43 show the Pearson coefficient for G-REALM and DAHITI separately for the different regions. The number of common lakes in each region is indicated in the x-axis. The value of Pearson coefficient is very high in most cases, with some exceptions, particularly when compared to G-REALM dataset. The timeseries for lakes with low values of correlation coefficient are very noisy in both datasets. Lake Vattern (Sweden), (Figure 44, has the lowest Pearson coefficient value (-0.165). Lake Vattern is the second largest lake in Sweden and is currently being monitored by Sentinel-3A and Sentinel-6A missions. However, in most cases, as with Lake Grande Trois (Canada) for example, the Pearson coefficient value is higher than 0.9 (Figure 45).

When compared to DAHITI dataset, the lake with the lowest Pearson coefficient (0.58) is Lake Hotta (Canada, see Figure 44). It is a lake located at high latitude (64.96 N), with a complex landscape, surrounded by multiple water bodies (Figure 47).

Figure 42: Comparison of regional (x-axis) LWL estimates to G-REALM estimates, using the Pearson Coefficient (x-axis). The number of common lakes in each region is indicated in the x-axis in brackets.

Figure 43: Comparison of regional (x-axis) LWL estimates to DAHITI estimates, using the Pearson coefficient (y-axis). The number of common lakes in each region is indicated in the x-axis in brackets.

 

Figure 44: Lake Vatern (Sweden). Timeseries of Lake Water Level estimates from C3S and G-REALM.

 

Figure 45: Lake Grande Trois (Canada). Timeseries of Lake Water Level estimates from C3S and G-REALM.

 


Figure 46: Lake Hotta (Canada). Timeseries of Lake Water Level estimates from C3S and DAHITI.

 

Figure 47: Lake Hotta. Canada, 64.96N, -118.39E.

2.2.1.1.2. Unbiased Root Mean Square Error (URMSE)

The second estimated metric is the Unbiased Root Mean Square Error, which corresponds to the squared difference between unbiased timeseries. Figure 48 and Figure 49 show the URMSE value for G-REALM and DAHITI separately for the different regions. The median of the URMSE value is 10.72 cm for G-REALM and 10.77 cm for DAHITI for lakes from all eleven regions. Lake Sobradino in Brazil (Figure 50) shows an URMSE value of 63.23 cm with a difference between C3S and G-REALM timeseries varying over time. However, there is a good correlation between Lake Sobradino from C3S compared to in-situ data from ANA (see 2.2.1.2.1, Figure 54), with a Pearson coefficient close to one and URMSE value of 26.3 cm.

Compared to DAHITI dataset, the lake with the highest URMSE value, 187cm, was estimated for Lake Toktogul (Kyrgyzstan). The timeseries presented in Figure 51 shows that this value is caused by the outliers in DAHITI timeseries in 2003 and the differences in the estimation of low water levels in 2022 and 2023.

Figure 48: Comparison of regional (x-axis) LWL estimates to G-REALM estimates, using the URMSE. The number of common lakes in each region is indicated in the x-axis in brackets. 

Figure 49: Comparison of regional (x-axis) LWL estimates to DAHITI estimates, using the URMSE. The number of common lakes in each region is indicated in the x-axis in brackets. 

Figure 50: Lake Sobradino (Brazil). Timeseries of Lake Water Level estimates from C3S and G-REALM.

 


Figure 51: Lake Toktogul (Kyrgyzstan). Timeseries of Lake Water Level estimates from C3S and DAHITI.

2.2.1.1.3. Bias

This metric indicates the difference of the reference for the water level estimation. Although we are mainly interested in the precision of the water level estimate, this information may be useful for studies on the accuracy of the estimate. Figure 52 and Figure 53 show the bias for each region for G REALM and DAHITI datasets.

Figure 52: Comparison of regional (x-axis) LWL estimates to G-REALM estimates, using the Bias as a measure. The number of common lakes in each region is indicated in the x-axis in brackets.

Figure 53: Comparison of regional (x-axis) LWL estimates to DAHITI estimates, using the Bias as a measure. The number of common lakes in each region is indicated in the x-axis in brackets.

2.2.1.2. In-situ products

The in-situ investigations compared LWL estimates available through the C3S service, to in-situ measurements of water level for water bodies monitored and measured by various agencies around the world. Here, they are presented in groups of water bodies, grouped by data provider. The information of the bias for in-situ data comparison does not provide useful information given that different surface references are used in the in-situ datasets.

2.2.1.2.1. Agencia Nacional de Aguas e Saneamiento Basico (ANA)

The National Water Agency of Brazil provides information on in-situ water level measurements for reservoirs⁸.  The Pearson correlation coefficients between C3S data and these measurements are shown in Table 5, and it is very high: greater than 0.9.

Table 5: C3S ANA Indicators. Pearson correlation coefficients between C3S LWL data and in-situ water level measurements in reservoirs monitored by the Water Agency of Brazil (Agencia Nacional de Aguas e Saneamiento Basico, ANA).

Figure 54: Comparison between C3S LWL data and in-situ water level measurements for Lake Sobradino (Brazil). Timeseries of Lake Water Level estimates from C3S and data provided by ANA.

2.2.1.2.2. Hidricos Argentina

The information concerning the daily historical variation on the Water Level for several in-situ stations in Argentina was obtained online from the “Base de Datos Hidrologica Integrada” (BDHI)⁹. Three lakes with daily data in this database are monitored by the C3S service. Pearson coefficients, URMSE values and the number of observations used for estimating the metrics for these lakes are indicated in Table 6.

Table 6: Comparisons between C3S LWL data and in-situ water level measurements for Argentinian lakes, using data provided by Hidricos Argentina. Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL data and in-situ water level measurements are shown for lakes available in the National Water Information System of Argentina (“Base de Datos Hidrologica Integrada, BDHI”).

The lake with the lowest Pearson correlation coefficient is Cochrane Lake. Figure 55 shows the timeseries from C3S and Hidricos Argentina. A change over the period 2020-mid-2021 is observed in the timeseries of in-situ data, which explains the lower correlation value.

 

Figure 55: Comparison between C3S LWL data and in-situ water level measurements for Lake Cochrane (Argentina), using data provided by the National Water Information System of Argentina ("Base de Datos Hidrologica Integrada, BDHI").

2.2.1.2.3. U.S. Geological Survey

The U.S. Geological Survey (USGS) provides information on water resources data collected mainly in the U.S. The USGS investigates the occurrence, quantity, quality, distribution, and movement of surface and underground waters, and disseminates the data to the public, state and local governments, public and private utilities, and other Federal agencies involved with managing the U.S. water resources. This report contains the comparison with a rigorous selection of in-situ stations defined as lakes in the USGS dataset. Three lakes were compared, all of them with high Pearson correlation coefficient (higher than 0.8) and low URMSE values (lower than 10 cm). the values of the different metrics for those lakes are indicated in Table 7. The Figure 56 shows the comparison of the timeseries for Lake Michigan (USA). 

Table 7: Comparisons between C3S LWL data and in-situ water level measurements for lakes whose measurements are made available by the U.S Geological Survey (USGS).

Figure 56: Comparison between C3S LWL data and in-situ water level measurements for Lake Michigan (USA), using data provided by USGS.

2.2.1.2.4. Water Office of Canada

In-situ daily data for Canadian lakes is freely available through the Water Office of Canada (Table 2). Several in-situ stations may be available for a single lake and a bias between them may exists and, for some of them, the reference for the water level measurements varies over time. To avoid the problems associated with these reference changes, the comparison was performed on the water level variation. 

Table 8: Comparisons between C3S LWL data and in-situ water level measurements for Canadian lakes, using data provided by Water Office of Canada. Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL data and in-situ water level measurements are shown for lakes available in the Water Office of Canada.

There is a good correlation for most of the lakes. The lowest value of the Pearson Coefficient, for Lake Winnipeg, is due to some outliers in the in-situ measurements as shown in Figure 57.

Figure 57: Comparison between C3S LWL data and in-situ water level measurements for Lake Winnipeg, using data provided by the Water Office of Canada.

The highest URMSE value was estimated for Lake Williston (Figure 58). In this case, there are some outliers in the C3S timeseries. 

Figure 58: Comparison between C3S LWL data and in-situ water level measurements for Lake Williston, using data provided by the Water Office of Canada.

2.2.1.2.5. Swiss Federal Office for the Environment (FOEN)

The Swiss Federal Office for the Environment (FOEN, see Table 2) implements environmental monitoring programs, and maintains various measurement networks. It operates and coordinates several water-related monitoring networks. Moreover, it monitors water level of rivers and lakes in Switzerland. Currently, two Switzerland lakes are monitored in the C3S Lakes program: Lake Bodensee and Lake Leman. Table 9 contains the values of the Pearson correlation coefficients, URMSE values, as well as the number of observations used to estimate those values, comparing the level variation timeseries from both lakes. 

Table 9: Comparisons between C3S LWL data and in-situ water level measurements for lakes whose measurements are made available by the Swiss Federal Office for the Environment (FOEN, see Table 3). Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL data and in-situ water level measurements are shown for lakes.

The correlations for both lakes are close to one, indicating a very good correlation level. Figure 59 shows the timeseries for Lake Leman from in-situ measurements and C3S estimates. Particularly, the increase in the water level in July 2021, due to an increase in rainfalls, was well detected by satellite observations.

Figure 59: Comparison between C3S LWL data and in-situ water level measurements for Lake Leman, using data provided by the Swiss Federal Office for the Environment.

2.2.2. Lake Water Level – Single track (LWL-S) Dataset

2.2.2.1. Altimetry products

Timeseries from C3S LWL-S v1.0 dataset were compared with G-REALM and DAHITI altimetric products. Only the measurements available in both (C3S and G-REALM or DAHITI) datasets on the same day were considered for the metrics computation.

2.2.2.1.1. Pearson coefficients

Figure 60 and Figure 62 illustrate the Pearson coefficients for G-REALM and DAHITI, respectively, for the regions with data availability in both (C3S and G-REALM or DAHITI) datasets. The number of lakes in each region is indicated by a number on each boxplot. 

Figure 60: Pearson coefficients calculated for lakes monitored at the same time by G-REALM and C3S LWL-S v1.0 dataset. The number of lakes in each region is indicated by a number on each boxplot.

Seventy lakes are monitored both in G-REALM and C3S LWL-S v1.0 datasets. Among them, 58 have a Pearson score higher than 0.8. Timeseries with low Pearson scores are very noisy, generally due to the presence of other water bodies that may contaminate the measurements. As an example of this effect, the Kitshomponshi Lake (Congo) timeseries (HydroLake¹⁰ ID: 181756), illustrated in Figure 61, has the lowest Pearson coefficient (0.16).

Figure 61: Kitshomponshi Lake (Congo). Comparison of timeseries LWL estimates from G-REALM and C3S.

 

Figure 62: Pearson coefficients calculated for lakes monitored at the same time by DAHITI and C3S LWL-S v1.0 dataset. The number of lakes in each region is indicated by a number on each boxplot.

Among the 25 lakes monitored by DAHITI and C3S LWL-S v1.0 dataset, 21 have a Pearson coefficient above 0.8. Two of them have a Pearson coefficient below 0.3 due to a low number of DAHITI altimetry measurements.

2.2.2.1.2. URMSE

Figure 63 and Figure 64 illustrate URMSE scores calculated for comparison with G-REALM and DAHITI products for the regions with availability of measurements in both (C3S and G-REALM or DAHITI) datasets. Median URMSE products equals to 21.7 cm and 34.8 cm for the comparison performed with G-REALM and DAHITI datasets, respectively. In the case of DAHITI, some URMSE values are high (above 50 cm) for the same reason as Pearson coefficients are low: the limited number of measurements of water levels at DAHITI for these lakes. In line with the Pearson coefficient results, Kitshomponshi Lake (Congo) shows the lowest URMSE value for comparison performed with G-REALM product. 

Figure 63: URMSE calculated for lakes monitored at the same time by G-REALM and C3S LWL-S v1.0 dataset. The number of lakes in each region is indicated by a number on each boxplot.

 

Figure 64: RMSE calculated for lakes monitored at the same time by DAHITI and C3S LWL-S v1.0 dataset. The number of lakes in each region is indicated by a number on each boxplot.

2.2.2.2. In-situ products

2.2.2.2.1. Hidricos Argentina

Information regarding the daily historical variation on the Water Level for several in-situ stations in Argentina was obtained online from the "Base de Datos Hidrologica Integrada" (BDHI). Two lakes with daily data in this database are monitored within the C3S LWL-S product. Table 10 presents the Pearson coefficients, URMSE values and the number of observations used for estimating the metrics for these lakes. Additionally, Figure 65 illustrates timeseries of Hidricos Argentina measurements and C3S LWL-S v1.0 data for the lake numbered 10465 in the HydroLake database (named Rio Laguna Setubal). This lake has higher URMSE value (0.43 m) from the two Argentinian lakes. Hidricos Argentina measurements show some discontinuities compared to the dates present in the C3S LWL-S v1.0 timeseries even though the acquisitions are available in both (C3S and G-REALM or DAHITI) datasets. This lack of data can lead to lower scores than those ideally obtained with a complete timeseries. The low number of available measurements within the comparison period was a reason for rejecting three other Hidricos Argentine stations. 

Table 10: Comparisons between C3S LWL-S data and in-situ water level measurements for Argentinian lakes, using data provided by Hidricos Argentina. Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL-S data and in-situ water level measurements are shown for lakes available in the National Water Information System of Argentina ("Base de Datos Hidrologica Integrada, BDHI"). 

Figure 65: Timeseries of water level anomalies from Hidricos Argentina measurements performed for the lake with the 10465 HydroLake ID (blue), compared to C3S LWL-S v1.0 corresponding timeseries (orange).

2.2.2.2.2. U.S. Geological Survey

The U.S. Geological Survey (USGS) provides information on water resources data collected mainly in the United States. They investigate the occurrence, quantity, quality, distribution, and movement of surface and underground waters, and disseminate the data to the public, state and local governments, public and private utilities, and other Federal agencies involved with managing the U.S. water resources. This report contains a comparison with a rigorous selection of in-situ stations defined as lakes in the USGS dataset. 

Eighteen lakes were compared, only eight lakes achieved a Pearson score above 0.8. This high number of lakes with an unfavorable score can be explained by the presence of consequent high variation frequency in retrieved timeseries due to retracking measurements and the geographical context around studied lakes. Furthermore, the mean URMSE value of these lakes is equal to 19.4 cm. The values of the different metrics for those lakes are indicated in Table 11. Figure 66 illustrates the timeseries obtained for the lake numbered 830 in the HydroLake dataset (named Lake Moultrie). The Pearson score is negative for this lake due to the presence of regular anomalies located during winter. Figure 67 illustrates the geographical situation of the lake. This lake is monitored by Sentinel-3A, whose measurements are acquired close to the lake shores, which complicates the retracking process (see the Algorithm Theoretical Basis Document (ATBD) [D3]). 

Table 11:  Comparisons between C3S LWL-S data and in-situ water level measurements for US lakes, using data provided by US Geological Survey lakes. Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL-S data and in-situ water level measurements are shown. 

Figure 66: Water level anomalies timeseries for the lake numbered 830 in the HydroLake database (Lake Moultrie, 33.3N, 80.0W).

 

Figure 67: Geographical context of the HydroLake ID numbered 830 (Lake Moultrie, 33.3N, 80.0W). The OLCT footprint is represented by the green rectangle.

2.2.2.3. SAIH

Sistemas Automáticos de Información Hidrológica (SAIH) provides information on water resources for reservoirs in Andalusia, Spain. The information is collected hourly but used daily to make comparisons. Daily products are distributed by SAIH as median value of hourly data. Three lakes are monitored in the C3S LWL-S v1.0 dataset with a low URMSE (below 20 cm) and high Pearson coefficients (near to 1.00) (see Table 12). Figure 68 illustrates the timeseries associated with the lake numbered 173656 in the HydroLake database (Aracena reservoir). 

Table 12: Comparisons between C3S LWL-S data and in-situ water level measurements for Spanish lakes, using data provided by SAIH lakes. Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL-S data and in-situ water level measurements are shown.

Figure 68: Water level anomalies timeseries for the lake numbered 173656 in the HydroLake database (Aracena reservoir, 37.9N, 6.5W).

2.2.2.3.1. Office of Public Work (Ireland)

The Office of Public Works (OPW) provides information on water resources for reservoirs in the Republic of Ireland. Two lakes are monitored in the C3S LWL-S dataset; Pearson scores and URMSE values are listed in Table 13. Figure 69 illustrates timeseries associated with the lake numbered 165579 (the Carrigadroihid reservoir). For this lake, the URMSE is above 20 cm and is caused by an anomaly that occurred at the end of the year 2021. 

Table 13: Comparisons between C3S LWL-S data and in-situ water level measurements for Irish lakes, using data provided by OPW lakes. Pearson correlation coefficients, URMSE and the number of common observations between C3S LWL-S data and in-situ water level measurements are shown.

Figure 69: Water level anomalies timeseries for the lake numbered 165579 in the HydroLake database (51.8N, 8.9W).

3. Application(s) specific assessments

Currently, no application(s) specific assessments have been undertaken for the C3S Lake Water Level (LWL) dataset V5.0 or Lake Water Level – Single track (LWL-S) V1.0.

4. Compliance with user requirements

The requirements for the C3S LWL are described in the 2023 Target Requirements and Gap Analysis document [D1].

Table 14. Compliance of the C3S LWL with user requirements.

The same requirements described in the 2023 Target Requirements and Gap Analysis Documents [D1] are associated with LWL-S products.

Table 15. Compliance of the C3S LWL-S with user requirements.

References

Lanczos, Cornelius (1988). Applied analysis. New York: Dover Publications. pp. 219–221. ISBN 0-486-65656-X. OCLC 17650089.

Cleveland, Robert B., Cleveland, William S., McRae, Jean E. and Terpenning, Irma. "STL: A Seasonal-Trend Decomposition Procedure Based on Loess (with Discussion)." Journal of Official Statistics 6 (1990): 3-73.

Messager, M.L., Lehner, B., Grill, G., Nedeva, I., Schmitt, O. (2016). Estimating the volume and age of water stored in global lakes using a geo-statistical approach. Nature Communications, 7: 13603. https://doi.org/10.1038/ncomms13603 (URL last accessed 22nd February 2024)

Schwatke, C., Dettmering, D., Bosch, W., and Seitz, F. (2015) DAHITI – an innovative approach for estimating water level time series over inland waters using multi-mission satellite altimetry, Hydrol. Earth Syst. Sci., 19, 4345-4364, https://doi.org/10.5194/hess-19-4345-2015 (URL last accessed 22nd February 2024), 2015.

ANNEX A. LWL Performance indicators

Related articles

• Page:
  LWL v5.0 and LWL-S v1.0: Product Quality Assessment Report (PQAR)
• Page:
  LWL v5.0 and LWL-S v1.0: Product User Guide and Specification (PUGS)
• Page:
  Lakes Service version 2023: Target Requirements and Gap Analysis Document (TRGAD)
• Page:
  LWL v5.0 and LWL-S v1.0: Algorithm Theoretical Basis Document (ATBD)
• Page:
  LWL v5.0 and LWL-S v1.0: System Quality Assurance Document (SQAD)