...
https://github.com/MeteoSwiss-APN/icon_data_processing_incubator/tree/main
In the context of the pilot project we are adapting it to make use of the FDB/polytope data technologies. This framework will be used to implement and deploy the use cases proposed in the pilot project, accessing from EWC the IFS operational data.
During this quarter we added functionality to the framework to:
- access data from FDB and/or polytope
- write results of the data processing back into grib and FDB/polytope
- implement application to re-project rotated lat-lon into geographical lat-lon (since it is the only grid supported at the moment by grib-jump)
Below we show an example of use of the framework that reads data from polytope:
The framework uses the earthkit (https://github.com/ecmwf/earthkit-data) as a framework for memory-only post-processing pipelines. Earthkit has seen substantial developments in the past months and is responsible for loading the grib objects into memory in Python (numpy compatible). It allows to efficiently process all the grib information in memory while retaining the original grib representation, therefore allowing to later write back the output as grib.
In the context of the pilot project we are adapting it to make use of the FDB/polytope data technologies. This framework will be used to implement and deploy the use cases proposed in the pilot project, accessing from EWC the IFS operational data.
During this quarter we added functionality to the framework to:
- access data from FDB and/or polytope
- write results of the data processing back into grib and FDB/polytope
- implement application to re-project rotated lat-lon into geographical lat-lon (since it is the only grid supported at the moment by grib-jump)
Below we show an example of use of the framework that reads data from polytope:
Code Block | ||||
---|---|---|---|---|
| ||||
def compute_ | ||||
Code Block | ||||
| ||||
def compute_echo_top(ref_time: dt.datetime, lead_time: int): request = mars.Request( ("HHL", "DBZ"), date=ref_time.strftime("%Y%m%d"), time=ref_time.strftime("%H00"), expver="0001", levelist=tuple(range(1, 82)), number=tuple(range(11)), step=lead_time, levtype=mars.LevType.MODEL_LEVEL, model=mars.Model.ICON_CH1_EPS, stream=mars.Stream.ENS_FORECAST, type=mars.Type.ENS_MEMBER, ) ds = mch_model_data.get(request, ref_param_for_grid="HHL") client = get_client() # Calculate ECHOTOPinM hfl = destagger(ds["HHL"], "z") echo_top = interpolate_k2any(hfl, "high_fold", ds["DBZ"], [15.0], hfl) echo_top.attrs |= metadata.override(echo_top.message, shortName="ECHOTOPinM") with data_source.cosmo_grib_defs(): client.to_fdb(echo_top) |
...
client.to_fdb(echo_top) |
These examples of use will see some consolidation work and will be released in the coming months.
Future work
The plan for the next quarter are:
- Further development and consolidation of the technologies and components already mentioned in this quarter.
- Start deploying first examples of the Flexpart showcase in EWC, using some of these components.
- Organize a GPU workshop with various key presentations from the community. The venue will be used to discuss and show the progress and efforts to port NWP and climate models and post-processing to GPUs. It will give an overview of the various approaches and technologies used.
- Continue the discussion around how to integrate GPU computing in the post-processing toolchain.