...
This is helping ECMWF consolidate the new developments of the FDB remote and debug some open issues. The most recent developments will be soon released (once the new functionality is stable). At the moment, we are using the following branch of FDB.
FDB performance
In order to evaluate the technology, it was important to characterize the performance obtained while retrieving data from FDB, using typical data access patterns of the downstream or post-processing applications.
We installed a complete FDB instance (in the CSCS lustre filesystem) at MeteoSwiss that holds the daily operational generated data. Based on that instance, we performance a series of benchmarking experiments.
Results depend on the access pattern (contiguity of data in a single data request, size of request, data being cache in the filesystem of the data servers, etc), but overall we obtained a high and satisfactory throughtput.
Results were summarized in the following notebook:
| Jupyter Viewer | ||
|---|---|---|
|