Contributors: L. Carrea (University of Reading), C. J. Merchant (University of Reading)
Issued by: L. Carrea, C. Merchant
Date: 19/07/2023
Ref: C3S2_312a_Lot4.WP3-SQAD-LK-v1_202301_LSWT_System_Quality_Assurance_i1.1
Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
General definitions
L2P – Geophysical variables derived from Level 1 source data on the Level 1 grid (typically the satellite swath projection). Ancillary data and metadata added following Group for High Resolution Sea Surface Temperature (GHRSST) Data Specification.
L3U – Level 3 Un-collated data are L2 data granules remapped to a regular latitude/longitude grid without combining observations from multiple source files. L3U files will typically be "sparse", corresponding to a single satellite orbit.
L3C – Level 3 Collated data are observations from a single instrument combined into a space-time grid. A typical L3C file may contain all the observations from a single instrument in a 24-hour period.
L3S – Level 3 Super-collated data are observations from more than one satellite that have been gridded together into a single grid-cell estimate, for those periods where more than one satellite data stream delivering the geophysical quantity has been available.
Brokered dataset – Dataset provided by another institution/initiative and not produced within this service.
Scope of the document
This document is the System Quality Assurance Document (SQAD) for Lake Surface Water Temperature (LSWT) in the Hydrology service of Copernicus Climate Change Service (C3S). It describes the systems used to provide the LSWT product to the C3S Climate Data Store (CDS), and all the contributions to the processing chain, interfaces to C3S, and to Research and Development (R&D) component, the hardware used in the processing chain, the procedures to implement new data cycles and to reprocess the products, and those applied in case of system failures or anticipated maintenance work, as well as information on user support.
Executive summary
The C3S Lake production system (C3S ECV LK) provides an operational lake surface water temperature (LSWT) and lake water level service, generating lake surface water temperature and lake water level climate datasets for a wide variety of users within the climate change community. Two groups support and maintain the systems that deliver data to the C3S for users - one dedicated to the lake surface water temperature and the other one to lake water level. The two parameters, lake surface water temperature and lake water level, have different systems (Technical Platforms or "processing chains") as their operation requires wholly different input data, ancillary data and algorithms. However, the concepts of both systems are similar.
This document is specifically devoted to the LSWT system, overviewing the people, hardware, software, and connections that ensure reliable delivery of high quality LSWT data to C3S users.
In terms of people, the LSWT group is subdivided into a science team, a system development team and an operation team. The science team directs the scientific evolution of the service and provides dedicated scientific support to the users. The system development team is responsible for the implementation of algorithm/product improvements, the maintenance and the validation of the processing chain. The operational team is responsible for the Operational Framework with designated responsibilities. These teams operate the operate and run the hardware, software, algorithm development and quality control that underpins data generation, and delivery.
This System Quality Assurance Document (SQAD) details the integral components of the C3S ECV LK – LSWT production system and their functions and interfaces (Section 1). The implementation procedure of the upgrade cycle is given in Section 2. Section 4 details procedures that are applied in case of system maintenance or failures and Section 5 provides information on the User support.
1. System overview
The Copernicus Climate Change Service (C3S) Lake production system (C3S ECV LK) is responsible for the operational provisioning of the Lake Essential Climate Variable (ECV) for the C3S. The mission of the system is the timely production of a consistent, long-term, multi-mission and multi-sensor global lake surface water temperature and lake water level data set complying with the defined key performance indicators (KPIs).This document is devoted to the system to generate the LSWT product.
1.1. System elements and interfaces
The system for generating data, delivering it to the C3S, and providing help and support to users is comprised of a number of different elements. These elements are comprised of people (e.g. the science team), hardware and software (e.g. the technical platform). They, and their interfaces with C3S are described in the following sections.
1.1.1. Components of the system
Integral components of the overall system for C3S Lake ECV LSWT are:
- the Organizational Framework consisting of
- the LSWT group at the University of Reading, subdivided into roles:
- the LSWT science team
- the LSWT system development team
- the LSWT operation team
- the LSWT group at the University of Reading, subdivided into roles:
- the Technical Platform for LSWT and its Operations Framework
- the C3S Processing Framework (CPF), and
- Data Management and Streams
These components, their functions and interfaces within the C3S ECV LK LWST production system will be discussed in the following sections. 1 (Organisational framework) is described in Section 1.1.2, while 2 to 4 are detailed in Sections 1.2 to 1.4 respectively.
1.1.2. Organizational Framework
Three teams at the University of Reading cooperate within the organizational framework in order to fulfil the mission objective of the system at the level of Lake ECV LSWT development, and data provision to users through the C3S:
- The Science Teams role is to take charge of the scientific validity of the LSWT products generated. The function of the Science Teams is to ensure that the scientific algorithms used are capable of producing a long-term consistent, multi-mission global LSWT products, in accordance with the defined data key performance indicators. The necessary Research & Development (R&D) has to be done externally to the project. The R&D was sourced from the European Space Agency (ESA) project Climate Change Initiative (CCI) LAKES1, for which the Phase 1 is now completed. It is hoped that future processing cycles will benefit from R&D funded within the auspices of ESA’s ongoing Climate Change Initiative. As the fundamental scientific approaches have been established, the C3S Science Team only focuses on developments such as determining parameters necessary to enable new sensors being brought into the service using existing scientific techniques.
- The System Development Teams role is to develop and maintain the implementation of the scientific algorithms based on the design of the Science Team, and integrate them into the processing baselines. They validate their implementation in accordance with the defined data key performance indicators and produce a new version of the systems that they deliver to the corresponding Operations Team.
- The Operations Teams role is to deliver the regular production of the LSWT products. With regards to that, the Operations Team undertakes provisioning and configuration of the technical platform, the setup and maintenance of the Operations Framework, and the provision of background User Support in response to queries from the C3S User Support facility.
These roles are defined separately to help structure the work conceptually. In terms of personnel, the teams overlap considerably. The interface between C3S and the teams happens mainly through Earth Observation Data Centre (EODC).
1.2. Hardware, supercomputers and cloud computing
1.2.1. Technical Platform and Operations Framework
The LSWT Technical Platform and Operations Framework is composed of a set of hardware and software components interacting with each other.
The Technical Platform is realized at the data and computation facility of the Centre for Environmental Data Analysis (CEDA), here referred to as “JASMIN”2, and consists of a dedicated project workspace with shared computational power that is used to generate the test and annual Climate Data Records (CDRs) for LSWT.
JASMIN is a “super-data-cluster”, half super-computer and half data-centre, which provides a high- performance storage fabric with a co-located batch compute cluster and private cloud. It is aimed primarily at the UK academic research community. JASMIN provides both "managed" and "unmanaged" cloud infrastructures.
- The "managed" infrastructure consists of high-performance disk storage situated alongside compute resources, both for virtualisation (i.e., hosting virtual machines) and batch compute (the LOTUS processing cluster3), which together make a managed analysis environment. The disk storage hosts the CEDA Archive as well as "Group Workspaces" which are allocations of high-performance disk shared between members of projects across a wide range of environmental science domains. In the “managed” part of the infrastructure, Science and Technology Facilities Council (STFC)-based system administrators are responsible for looking after the systems, installing known and trusted software and managing any outward-facing services, which are made available through the STFC/RAL (Rutherford Appleton Laboratory) firewall. These services include the CEDA data access services (Fast Transfer Protocol: FTP, Hypertext Transfer Protocol: HTTP, Open-source Project for a Network Data Access Protocol: OPeNDAP).
Within the managed infrastructure, compute demands are predominantly focused on LOTUS, the batch processing cluster, rather than individual virtual machines.
- In the "unmanaged" environment, the "Infrastructure as a Service" model of cloud provision is used to provide projects with their own environment in which they can host their own services and workflows. Projects (tenants) are provided with an allocation of storage, Central Processing Unit (CPU), memory and a range of network addresses which form a cloud "virtual organisation" in which they can build their own systems. The tenants provide their own system administrator(s) and maintain the virtual machines themselves, which sit on a portion of network address space assigned to Natural Environment Research Council (NERC). The tenants manage their own outward-facing services in this area of the network, which sits outside the STFC firewall.
Figure 1 illustrates the functional view of the JASMIN infrastructure. The C3S LSWT project is making use of a number of these parts within the project as a whole. For the brokered CDR datasets, the relevant parts are the CEDA archive and download services, shown in green. Data production for the Intermediate Climate Data Record (ICDR) further makes use of the CEDA batch compute (LOTUS) and scientific analysis and data transfer machines, as well as group workspace storage. Once short delay processing commences, additional services using the unmanaged cloud, may also be used.
Figure 1: Functional view of the JASMIN infrastructure (courtesy of CEDA), a dedicated project workspace with shared computational power that is used to generate the test and annual Climate Data Records (CDRs) for LSWT, showing the range of services provided. The dataset production makes use of the CEDA batch compute (LOTUS) and scientific analysis (sci) and data transfer machines (xfer and gridftp) and for the archive and download services the relevant parts are shown in green (Archive Tape, Archive, and CEDA Services).
The system described above generates the C3S annual extensions to the LSWT record. The same system was used to generate the brokered dataset from the project ESA CCI LAKES LSWT4 covering the earlier years of the CDR.
In terms of software and hardware interfaces that enable generation of the LWST data, these are shown in Figure 1. There are also some procedural connections / interfaces that ensure delivery of the generated data to the CDS and onwards to users. Once the annual extension of the LSWT record is finalized, the University of Reading delivers the LSWT data to the EODC via an sftp server. The EODC then provides the dataset to CDS via a dedicated web server for data provision.
The Operations Framework is based on source code that is version-controlled in Github5, holding all code and configurations needed to run the operational service. System operations is done by members of the Operations Team.
1.3. C3S Processing Framework
Source data used to generate the LSWT product (Figure 2) includes satellite-derived level 1 (radiance) products from space agencies and numerical weather prediction (NWP) files from European Center for Medium-range Weather Forecasts (ECMWF). The responsibility for collecting these data streams lies with CEDA, as part of the agreement of the LSWT service with CEDA. Other auxiliary data are the responsibility of the Science Team (since they embed algorithmic knowledge). The data readers are developed by the System Team in common between LSWT and the C3S Sea Surface Temperature (SST) service, since the data input streams are overlapping, and this ensures consistency.
Figure 2: Processing framework used to create C3S LSWT data which are delivered to C3S via EODC (see Section 1.2.1).
Processors implement the scientific retrieval methods at full resolution, creating full resolution intermediate outputs, which are compared with validation data. The collection of validation data updates is an annual task of the Science Team (no functional system equivalent to the SST International Comprehensive Ocean-Atmosphere Data Set (ICOADS)6 system exists for LSWT, so this is done via personal networks). The University of Reading delivers the LSWT data to the EODC which hosts a web server for data provision to the CDS.
1.4. Data management and Streams
The input data streams for LSWT-4.5 (C3S v4.0 CDR) are listed in Table 1. In future cycles: i) further streams may be added, the next priority being MetOp Advanced Very-High Resolution Radiometer (AVHRR); (ii) some data may be sourced from alternative routes in the future (e.g., National Oceanic and Atmospheric Administration: NOAA instead of European Organisation for the Exploitation of Meteorological Satellite: EUMETSAT); (iv) and ERA-Interim has been superseded partly for the LSWT-4.5 extension.
Table 1: Input data streams for LSWT-4.5 (C3S v4.0 CDR).
Satellite data stream | Origin | Responsibility | Collection schedule |
Sentinel3A & 3B | ESA | CEDA | Typically 1 or 2 days behind |
Terra MODIS | NOAA | CEDA | Typically 1 or 2 days behind |
NWP data stream | Origin | Responsibility | Collection schedule |
ERA5 | ECMWF | CEDA | Typically 4 months behind |
Validation data stream | Origin | Responsibility | Collection schedule |
Sustained and occasional in situ collections | Network of direct contacts | Science team | Additional data are solicited annually in November/December |
2. Upgrade cycle implementation procedure
2.1. Technical Platform and Operations Framework upgrades
A production of an updated CDR product is expected to occur once per year. There is an annual cycle of upgrades to the processing chain and operational frameworks, linked to the annual production cycle:
- End of January: CDR products are generated and disseminated.
- February: Issues to be addressed for the next cycle are identified in part by validation activities and in part by gap analysis, and are diagnosed.
- August: Design and initial testing of developments culminate in generation of the test CDR product.
- January: Verification of the test results determines the full implementation of the upgrades to the processing chain and documentation in time for the next CDR generation.
2.2. Scientific developments / Processing Framework upgrades
The baseline service for LSWT is to provide consistent LSWT CDRs using the existing algorithms, with ad hoc improvements to those as required. These improvements are implemented as system upgrades. This is because only fundamental R&D will lead to new algorithms and is beyond the scope of the service. Improvements to algorithm parameters and adaptation to newly introduced sensor data streams are in scope for the service, so upgrades will be made, triggered by the availability of revised or additional auxiliary or input data streams.
It is hoped, but is not yet confirmed, that more fundamental R&D for LSWT will be pursued within the ESA Climate Change Initiative. Products and algorithms updates shall only be integrated into C3S after the verification and validation of new CCI LAKES data/methods are performed successfully. Implementation in the C3S of the algorithm updates will be performed in parallel to the operational service at least two months before a new CDR production cycle is started (i.e., before mid-October). This would trigger a major version number change (in the public-facing versioning nomenclature), to make clear with which brokered data set (ESA CCI LSWT-4.5, etc) the CDR generated annually under C3S can be seamlessly used.
Communication about the release of a new product version will be started three months prior to release in the Climate Data Store (CDS).
3. Procedures for reprocessing CDR's (not applicable)
Reprocessing of the long-term LSWT CDR is not expected within the C3S service. Reprocessing is expected within the framework of the ESA CCI programme for the Lakes ECV.
4. System maintenance and system failures
Any issues or outages, whether planned or unplanned, will be documented in the Quarterly Report.
4.1. Dissemination failures
Product dissemination is addressed centrally by EODC for the Hydrology service, presented in Quarterly Status reports and is not applicable for this document.
In terms of ensuring EODC can be passed products in a timely fashion, planned outages of the CEDA archive are kept to a minimum, and are publicised in advance to all CEDA users via email, via the CEDA news feed on their website7 and Twitter/Facebook channels, enabling planning.
4.2. System outages
In the case of unforeseen system failures or other problems affecting data delivery to the C3S the following procedure will be applied:
1. On notification of a problem, internal CEDA procedures will be invoked to investigate and resolve the issue.
2. EODC will be notified by the LSWT Operations Team, who in turn alert the C3S CDS.
It should be noted that support for the CEDA and JASMIN infrastructures and for data delivery issues is only available during standard UK working hours.
As noted in Section 1.2, LSWT data are delivered first to the EODC and from there are made available for the CDS to ingest. While a production of an updated CDR product is expected to occur once per year (Section 2), issues in dataset generation do not impact users ability to access existing data on the CDS.
4.3. System performance
In case of other issues affecting system performance (such as delaying the annual processing), these will be investigated and reported to the CDS as per the case of the system outages above. In the case of complex issues which may take longer to resolve, the progress will be reported regularly to C3S and will be included in the Quarterly Reports.
4.4. System failures
Failures may arise abruptly any time, but considering the annual generation of the products, failures can be treated without any major delay in the production, except during the critical generation window (December / January). Notification of EODC and the CDS will be as per the case of the system outages above.
4.5. Loss of key staff
At a first response key staff can be replaced by qualified colleagues from within the partner organisation. If this is not possible then the staff will be replaced by external candidates. In the meantime, tasks may be distributed among partners to ensure a continuity of project progress.
4.6. Input data stream failure
The product relies upon a suite of earth observation sensors. The loss of one or a few input products will degrade the quality, and not critically stop the production of the CDS product. Only the loss of all input products would result in the inability to generate a product.
5. User support
A dedicated service desk has been set up, the Copernicus User Support (CUS) team, which provides support to users of the Copernicus Atmosphere Monitoring Service (CAMS) and C3S services at ECMWF. All enquiries about the LSWT dataset must be submitted through the service desk where it will be dealt with by appropriate persons.
C3S operates a dedicated user support portal8 where customers can submit enquiries using a form (split into “Get our products”, “Licence and invoice”, “Scientific or technical question about our products” and “Problem receiving or accessing our products”). The information from this form is passed to the CUS. Once submitted, the user may add comments or further information to the issue, including responding to questions / requests for additional information from the support team.
In addition to submitting enquires through the portal a knowledge base9 is available to users which can be searched for information.
The Lakes service has a team account with the Copernicus User Support Jira Service Desk System, to provide level 2 user support, i.e. to answer enquiries specific to their products, by direct interaction with the user through the Jira helpdesk. Once a request is sent, the Copernicus User Support (CUS) Service team at ECMWF will handle the requests within 8 hours (level 1).
For any scientific and special enquiries that cannot be answered by the CUS team at ECMWF or addressed to the Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2). Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user.
In case of specific scientific issues, the enquiries will be channelled to the ECV and data specialist of the C3S2_312a_Lot4 project (under which the LSWT dataset is made available) and should be resolved within 3 working weeks (target 85%).
In each quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets in the Quarterly Report.
