...
The main outcomes of this solution are easy access to data and performance improvements.
Easy Access to Data
Example:
|
Performance Improvements
Polytope's feature extraction functionality significantly improves data access efficiency. Below is an example demonstrating the reduced data transfer and storage on the user side:
Example: A user wants to calculate the 50th percentile for all ensemble forecasts in March at an airport's 10m wind speed field. The raw model files amount to 90 TB. Using FDB to request the necessary data reduces this to 60 GB, as FDB indexes the data to retrieve only the wind speed field. After computing the median forecast, the data size further shrinks to 6 GB. This 6 GB file is stored in FDB for potential use by others. Finally, a user interested in just one specific point can employ feature extraction in Polytope, retrieving only the data for the airport, resulting in a data transfer of approximately 6 kilobytes.
|
The development and implementation of these functionalities were made possible through the contributions of Emanuele Danovaro, Christopher Bradley, Mathilde Leuridan, James Hawkes, and Tiago Quintino from ECMWF.
...
We preserve the original GRIB message template as attributes within the xarray.DataArray (e.g. precipitation field shown below). This ensures that the structural integrity of the GRIB data is maintained throughout the workflow.
Example: Xarray object of the precipitation field with the GRIB message included in its attributes:
|
Ensuring Metadata Integrity:
We have implemented a collection of operators that update and maintain metadata accuracy. This guarantees that any modifications to the data are reflected appropriately in the metadata.
Example: The destagger operator updates the field's GRIB message after the field has been destaggered:
|
These contributions enable users to write xarray objects to GRIB2 files, a functionality that was previously unavailable. This enhancement is crucial for workflows that require the creation and manipulation of GRIB2 data.
...
ICON's Operational GPU Integration by Xavier Lapillonne, MeteoSwiss: We explored rewriting of NWP models for GPU efficiency, illustrated by the recently operational ICON project at MeteoSwiss. We learned about the integration of ICON into the new ALPS High-Performance Computing Platform at CSCS, addressing operationalization and maintenance challenges.
Performance and Adaptability with Domain Specific Language (DSLs) by Christoph Müller, MeteoSwiss: Highlights of the shift to Python-based user codes using DSLs like GT4Py, emphasizing performance portability and adaptability.
Starting anew with the Development of Momentum® Weather and Climate Model by Iva Kavcic, Met Office: We explored new horizons with the Next Generation Modelling System Programme from the Met Office, paving the way for a new dynamical core and software infrastructure in weather and climate modelling, with plans for implementation from the mid-2020s onwards.
NWP Models on AMD GPUs by Bentorey Hernandez Cruz, ECMWF: Details of the ongoing development of NWP models on AMD GPUs within the DestinE project on the LUMI supercomputer, addressing the challenges and progress in utilizing AMD's architecture for weather and climate modeling.
| Info |
|---|
| A recording of the webinar is available for download here (Password: GPU_webinar_pilot24). |
...