...

At the same time, the pilot aimed to assess the impact of these changes on Member States’ workflows and gather insights that could help refine ECMWF’s own systems , ensuring they to better support future developments and align with the meet Member States’ evolving needs of its users.

Key Objectives include:

1. Developing and deploying open-source blueprints for efficient data access and processing, leveraging ECMWF technologies like Aviso for workflow automation, FDB for semantic access to data, Earthkit for managing GRIB data in Python, and Polytope for efficient data retrieval.
2. Facilitating ECMWF product integration with the European Weather Cloud (EWC) enabling Member States to run workflows and pre-process IFS data before dissemination.
3. Establishing a framework for collaboration on new data processing technologies, such as GPU acceleration and domain-specific languages.
4. Exploring funding opportunities and long-term sustainability, following unsuccessful Horizon Europe funding attempt.

...

To refine and validate our approach, we collaborated with key partners actively working with Flexpart on the EWC and Atos. The Royal Meteorological Institute (RMI) of Belgium, which operates a web application for Flexpart simulations on the EWC, had encountered challenges with data retrieval. They expressed faced performance issues when retrieving IFS forecast data via the MARS client. They showed strong interest in our approach using Polytope to optimize data access.using Polytope as a faster alternative. However, this was initially limited by the absence of essential input fields on Polytope at ECMWF, which at the time only hosted a subset of parameters on model level (ml). Following several discussions, ECMWF agreed to make a limited set of additional variables available on Polytope by mid to end of May 2025. While not yet officially confirmed, ECMWF is also exploring a longer-term solution through a potential Data Bridge, hosted on SwissTwin at the Swiss National Supercomputing Centre (CSCS), which could eventually provide broader access to the for all Member States.

We also engaged with partners at Additionally, we engaged with partners of the University of Vienna, the main developers of Flexpart, to exchange insights on our activities and share experiences working on the European Weather Cloud. They shared similar limitations, particularly in reported similar challenges—especially around workflow automation, data retrieval, and infrastructure scalability. They were also interested in deploying a service that runs scalability—and expressed interest in running Flexpart on Kubernetes within the EWC but faced the same constraints. These limitations, further detailed in the Key Outcomes and Identified Limitations section below, underscore the need for improvements in resource management and infrastructure flexibility within the EWC.

Key Outcomes and Identified Limitations

• Cloud Resource Management: The workflow currently runs on a continuously active virtual machine instead of dynamically provisioning resources (e.g., via Kubernetes). This is inefficient, but EWC is expected to support short-lived job execution by June 2025.

...

. These discussions provided valuable context and reinforced the need for more flexible, scalable infrastructure across the platform.

The pilot enabled us to identify key limitations but also helped trigger improvements through ongoing dialogue with ECMWF and the EWC team:

Key Outcomes:

• Data Access via Polytope: Several required IFS fields were initially unavailable on Polytope. After repeated exchanges, ECMWF agreed to make a limited set of missing fields available on ECMWF Polytope by the end of May 2025. A broader solution is under consideration through the development of a Data Bridge on SwissTwin, though this remains tentative

...

• . Estimated timeline: End of Q2 2025.
• Infrastructure as Code (IaC): Initially, deployment within the EWC required manual setup, which was time-consuming

...

• and prone to inconsistencies. Throughout the pilot, we engaged in extensive discussions with the EWC support team to highlight the need for an automated and reproducible deployment process. Following these discussions, Terraform-based IaC

...

• support has been made available in the EWC since mid-March 2025. While it is not yet integrated into our pipeline, its availability reflects how the pilot helped identify and prioritize this requirement.

• Container & IaC Repository: No Currently, there is no shared repository exists within the EWC for container images and IaC configurations. We are in discussions with the EWC team about this limitation, as a dedicated registry would significantly enhance collaboration and consistency. In the meantime, DockerHub remains a viable alternative for managing and sharing these resources.

• Cloud Resource Management: The workflow currently runs on a continuously active virtual machine instead of dynamically provisioning resources (e.g., via Kubernetes). This is inefficient. EWC is expected to support short-lived job execution by June 2025, and we are in contact with the EWC team to follow up on progress and to be ready to test the functionality as soon as it becomes available While DockerHub is an alternative, a dedicated EWC registry would enhance collaboration.

Remaining Actions Within the Project Timeline Through End of 2025

...

The preprocessing step is essential for preparing ECMWF meteorological fields as input for Flexpart. It involves performing various operations on the input data, such as deaggregating precipitation values and other transformations required for accurate atmospheric transport modeling. With Earthkit-Data, these fields can be efficiently read in GRIB format and converted into Xarray datasets, enabling powerful multi-dimensional operations that enhance readability, flexibility, and efficiency.

The current repository is private; however, access can be granted upon request for those interested.

Key Outcomes

• Improved Readability & Maintainability: Transitioning from Fortran to Python enhances code clarity and flexibility.
• Simplified GRIB Handling: Earthkit-Data simplifies ECMWF data processing

...

▢ Deployment Improvements – Streamline image building and deployment, enable rollback capabilities, maintain a stable production version while testing new releases (DEPL), automate container build chains, and introduce alerts for failed deployments.

Advancing Environmental Data Retrieval (EDR) Integration

Remaining Actions Within the Project Timeline Through End of 2025 Potential Actions in Case of Project Extension

A key objective of the extension is the development of Environmental Data Retrieval (EDR) based on Open Geospatial Consortium (OGC) Standards, in collaboration with UK Met Office, a leading contributor to its development. EDR aims to standardize meteorological data retrieval within the OGC framework, improving interoperability and accessibility across geospatial and forecasting applications.

While EDR is an established standard, further adaptation is needed for its effective use in meteorology. The extension year would enable collaboration with UK Met Office to:

• Enhance support for meteorological data formats, such as GRIB, NetCDF, and BUFR, to improve alignment with existing standards.
• Address current limitations, such as the lack of support for ensemble (ENS) data, which is crucial for probabilistic forecasting. EDR does not yet define a standardized method for requesting or representing ensemble datasets, limiting its applicability for meteorological use cases.
• Leverage MeteoSwiss expertise in data retrieval APIs, drawing from its experience in opening its data this year, to contribute to refining the standard and improving accessibility.

Facilitating Knowledge Sharing through Webinars

...