...
In this document we describe the progress and contributions made to different activities in preparation of the deployment of show cases in EWC.
Personnel Involved:
Victoria Cherkas , Christian Kanesan, Stefan Friedli (MeteoSwiss) contributed in-kind to all the tasks described below.
The hiring of personnel with the resources of the project was completed and Nina Burgdorfer will start contract on 15.03.2024
Data Technologies Workshop, ECMWF - MeteoSwiss, 6-10.11.2023
...
The topics discussed during the workshop cover covered among others:
- Data processing frameworks in python:
numpy/xarray: should we embrace rich functionality of frameworks like xarray or use pure numpy API solutions for compatibility?
- how to write data back to grib and other data formats.
Metadata and grid information in the framework.
Evaluate and learn from typical member state operators and grids for data processing.
Dask and task scheduling: How to parallelize the dags.
Performance evaluations: dask, numba, GPU.
Event driven processing.
- FDB:
- How to deploy a FDB server in order to deploy data out of the HPC center.
- Learn from operational aspects of deployments of FDB at ECMWF.
- Analysis of FDB performance for the MeteoSwiss data deployment
- polytope:
- Design of the data bridge design using polytope serving FDB data.
- MARS language: for COSMO/ICON data of the MeteoSwiss operational data.
...
It will be used in the pilot project in order to retrieve and access data semantically instead of the traditional (grib) file based approach, employed still in many operational environments of NMHS.
FDB is implements a field database optimized for HPC data centers with a (lustre) distributed file system and adds a Python frontend to facilitie facilitate the data access of meteorological fields in Python.
...