During this quarter we focused on the preparation of software components and technology readines needed to start deployment the show cases described in the Statement of Work.

In this document we describe the progress and contributions made to different activities in preparation of the deployment of show cases in EWC.

Personnel Involved: 

Victoria Cherkas ,  Christian Kanesan, Stefan Friedli (MeteoSwiss) contributed in-kind to all the tasks described below. 

The hiring of personnel with the resources of the project was completed and Nina Burgdorfer will start contract on 15.03.2024

Data Technologies Workshop, ECMWF - MeteoSwiss, 6-10.11.2023

In order to kick-off the collaboration around data technologies, a workshop was organized between MeteoSwiss and ECMWF at the ECMWF center (Reading).

Participation: Christian Kanesan (MeteoSwiss), Victoria Cherkas (MeteoSwiss), Petra Baumann (MeteoSwiss), Carlos Osuna (MeteoSwiss), Stefan Friedli (MeteoSwiss - remote), Milos Belic (MeteoSwiss - remote), Emanuele Danovaro (ECMWF), Christopher Bradley (ECMWF),  Simon Smart (ECMWF), James Hawkes (ECMWF), Tiago Quintino (ECMWF), Sandor Kertesz (ECMWF)

The goals were:

• get-to-know event for the team of developers.
• define goals of the collaboration and exchange of development plans. 
• understanding of technology, use cases and define roadmap for the collaboration.

The topics discussed during the workshop covered among others: 

• Data processing frameworks in python:

  • numpy/xarray: should we embrace rich functionality of frameworks like xarray or use pure numpy API solutions for compatibility?

  • how to write data back to grib and other data formats.

  • Metadata and grid information in the framework.

  • Evaluate and learn from typical member state operators and grids for data processing.

  • Dask and task scheduling: How to parallelize the dags.

  • Performance evaluations: dask, numba, GPU.

  • Event driven processing.

• FDB: 
  • How to deploy a FDB server in order to deploy data out of the HPC center.
  • Learn from operational aspects of deployments of FDB at ECMWF.
  • Analysis of FDB performance for the MeteoSwiss data deployment
• polytope:
  
  • Design of the data bridge using polytope serving FDB data.
• MARS language: for COSMO/ICON data of the MeteoSwiss operational data.

Conclusions and results of the different activities are summarized in the following sections of the quarterly report.

FDB

FDB is an essential component for the pilot project. FDB is a domain-specific object store developed at ECMWF for storing, indexing and retrieving GRIB data. 

It will be used in the pilot project in order to retrieve and access data semantically instead of the traditional (grib) file based approach, employed still in many operational environments of NMHS.

FDB implements a field database optimized for HPC data centers with a (lustre) distributed file system and adds a Python frontend to facilitate the data access of meteorological fields in Python. 

The following shows an example of how to retrieve a full hypercube of ensemble data for two fields (height and DBZ) from COSMO data: 

request = mars.Request(
        ("HHL", "DBZ"),
        date=ref_time.strftime("%Y%m%d"),
        time=ref_time.strftime("%H00"),
        expver="0001",
        levelist=tuple(range(1, 82)),
        number=tuple(range(11)),
        step=lead_time,
        levtype=mars.LevType.MODEL_LEVEL,
        model=mars.Model.COSMO_1E,
        stream=mars.Stream.ENS_FORECAST,
        type=mars.Type.ENS_MEMBER,
    )
ds = model_data.get(request, ref_param_for_grid="HHL")

Remote FDB

An important component of the FDB framework is the FDB remote. This new development of the FDB family allows to serve FDB data to requests that do not have access to the Lustre based deployment of FDB. 

The FDB remote is implemented as a set of services, including a catalogue (for querying data) and a store (that serves data requests).

A typical use case that needs the deployment of the FDB remote is the access of ECMWF production data from the EWC environment.

One of the architecture designs of pilot project is based on access to ECMWF production directly from EWC (without the need to setup and maintain dissemination streams). However, EWC is not in the same network as the HPC production of ECMWF. Applications running on EWC can not acccess the main FDB of the HPC center for retrieving data. FDB remote will be setup, such that any application in EWC will be able to request data to the server.

A simplified design of this deployment is shown in the following diagram: 

In the pilot project, we have deployed a similar architecture with a FDB instance at MeteoSwiss (in the MeteoSwiss operational dedicated HPC system at CSCS) in order to test this new server technology, together with a polytope deployment. 

In this case, we are serving data (COSMO/ICON) generated daily by the MeteoSwiss operational suite. 

This is helping ECMWF consolidate the new developments of the FDB remote and debug some open issues. The most recent developments will be soon released (once the new functionality is stable). At the moment, we are using the following branch of FDB. 

FDB performance

In order to evaluate the technology, it was important to characterize the performance obtained while retrieving data from FDB, using typical data access patterns of the downstream or post-processing applications. 

 We installed a complete FDB instance (in the CSCS lustre filesystem) at MeteoSwiss that holds the daily operational generated data. Based on that instance, we performance a series of benchmarking experiments. 

Results depend on the access pattern (contiguity of data in a single data request, size of request, data being cache in the filesystem of the data servers, etc), but overall we obtained a high and satisfactory throughtput. 

Results were summarized in the following notebook:

Polytope