During this quarter we focused on the preparation of software components and technology readines readines needed to start deployment the show cases described in the Statement of Work.

In this document we describe the progress made.

Data Technologies Workshop, ECMWF - MeteoSwiss, 6-10.11.2023

In order to kick-off the collaboration around data technologies, a workshop was organized between MeteoSwiss and ECMWF at the ECMWF center (Reading).

Participation: Christian Kanesan (MeteoSwiss), Victoria Cherkas (MeteoSwiss) Petra Baumann (MeteoSwiss), Stefan Friedli (MeteoSwiss - remote), Milos Belic (MeteoSwiss - remote), Emanuele Danovaro (ECMWF), Christopher Bradley (ECMWF),  Simon Smart (ECMWF), James Hawkes (ECMWF), Tiago Quintino (ECMWF), Sandor Kertesz (ECMWF)

The goals were:

• get-to-know event for the team of developers.
• define goals of the collaboration and exchange of development plans. 
• understanding of technology, use cases and define roadmap for the collaboration.

The topics discussed during the workshop cover among others: 

• Data processing frameworks in python:

  • numpy/xarray: should we embrace rich functionality of frameworks like xarray or use pure numpy API solutions for compatibility?

  • how to write data back to grib and other data formats.

  • Metadata and grid information in the framework.

  • Evaluate and learn from typical member state operators and grids for data processing.

  • Dask and task scheduling: How to parallelize the dags.

  • Performance evaluations: dask, numba, GPU.

  • Event driven processing.

• FDB: 
  • How to deploy a FDB server in order to deploy data out of the HPC center.
  • Learn from operational aspects of deployments of FDB at ECMWF.
  • Analysis of FDB performance for the MeteoSwiss data deployment
• polytope:
  
  • Design of the data bridge design using polytope serving FDB data.
• MARS language: for COSMO/ICON data of the MeteoSwiss operational data.

FDB