During this quarter we focused on the preparation of software components and technology readines readiness needed to start deployment the show cases described in the the Statement of Work.
In this document we describe the progress and contributions made to different activities in preparation of the deployment of show cases in EWC.
...
| Code Block | ||
|---|---|---|
| ||
request = mars.Request(
("HHL", "DBZ"),
date=ref_time.strftime("%Y%m%d"),
time=ref_time.strftime("%H00"),
expver="0001",
levelist=tuple(range(1, 82)),
number=tuple(range(11)),
step=lead_time,
levtype=mars.LevType.MODEL_LEVEL,
model=mars.Model.COSMO_1E,
stream=mars.Stream.ENS_FORECAST,
type=mars.Type.ENS_MEMBER,
)
ds = model_data.get(request, ref_param_for_grid="HHL") |
Remote FDB
During this quarter, the project deployed an instance of FDB at CSCS for COSMO/ICON data. The operational NWP COSMO/ICON data is directly pushed into the FDB instance so that applications and post-processing (mostly in Python) can access data semantically as with the example shown above. This work required adaptations and extensions of the MARS language used to index semantically the grib data in order to support the COSMO/ICON operational data schemes. It provided valuable insights on how a NMHS must modify the MARS language to be able to use all the FDB based data technologies on its own operational data.
Remote FDB
As described above, FDB is a field database based on a distributed (e.g. Lustre) file system. However, in various use cases we would like to access the FDB data from a network that does not have access to the operational filesystem. Therefore, an An important component of the FDB framework is the the FDB remote. This new development of the FDB family software stack allows to serve FDB data to requests that do not have access to the Lustre based deployment of FDB.
...
One of the architecture designs of pilot project is based on access to ECMWF production directly from EWC (without the need to setup and maintain dissemination streams). However, EWC is not in the same network as the HPC production of ECMWF. Applications running on EWC can not acccess access the main FDB of the HPC center for retrieving data. FDB remote will be setup, such that any application in EWC will be able to request data to the server.
...
This is helping ECMWF consolidate the new developments of the FDB remote and debug some open issues. The most recent developments will be soon released (once the new functionality is stable). At the moment, we are using the following branch of FDB.
During the development phase regular meetings with the FDB developers were scheduled in order to work together, identify issues of instability and define actions to fix/improve the server. Out of this work we expect a stable version that can be used for the pilot project.
FDB performance
In order to evaluate the technology, it was important to characterize the performance obtained while retrieving data from FDB, using typical data access patterns of the downstream or post-processing applications of a NHMS.
We We installed a complete FDB instance (in the CSCS lustre filesystem) at MeteoSwiss that holds the daily operational generated data. Based on that instance, we performance a series of benchmarking experiments.
...