Title: Progress report for Pilot project on "Adaptation to Emerging Technologies"

Period: Q3 2024 – ECMWF PAC Meeting (30 April 25)

Summary

 This short report presents the progress and evaluation of the pilot project to date. It highlights key achievements, assesses the project's impact on collaboration, and offers reflections on the setup and lessons learned.

1. Project Objectives

The Adaptation to Emerging Technologies pilot project was launched to explore how ECMWF and its Member States can adapt to new technologies for accessing and processing data, developed by ECMWF within the Destination Earth project, and integrate these advancements into Numerical Weather Prediction (NWP) workflows. The project set out to demonstrate how these technologies could be leveraged to enhance operational data processing, ensuring seamless integration with ECMWF’s computing resources, including the European Weather Cloud (EWC) and its data infrastructure.

At the same time, the pilot aimed to assess the impact of these changes on Member States’ workflows and gather insights that could help refine ECMWF’s own systems, ensuring they better support future developments and align with the evolving needs of its users.

Key Objectives include:

1. Developing and deploying open-source blueprints for efficient data access and processing, leveraging ECMWF technologies like Aviso for workflow automation, FDB for semantic access to data, Earthkit for managing GRIB data in Python, and Polytope for efficient data retrieval.
2. Facilitating ECMWF product integration with the European Weather Cloud (EWC) enabling Member States to run workflows and pre-process IFS data before dissemination.
3. Establishing a framework for collaboration on new data processing technologies, such as GPU acceleration and domain-specific languages.
4. Exploring funding opportunities and long-term sustainability, following unsuccessful Horizon Europe funding attempt.

Two primary workstreams were defined:

1. Flexpart Dispersion Model – implementing an end-to-end workflow for the use of atmospheric dispersion simulations on the European Weather Cloud.
2. IFS Data Pre-processing – optimizing workflows for extracting, transforming, and aggregating forecast data for various applications.

The project set out to provide open, adaptable technological solutions while fostering knowledge exchange through workshops and regular collaboration meetings.

3.     Achievements and Outcomes

This pilot project demonstrates how emerging technologies can modernize data workflows for NWP while requiring adaptations from Member States to fully leverage ECMWF’s evolving data access tools. By implementing concrete use cases, we validated new approaches and provided valuable feedback that improved ECMWF’s own systems. The project focused on five key areas:

4.     Key Developments and Results

Flexpart Workflow Deployment on European Weather Cloud (EWC)

This use case created a blueprint that integrates the European Weather Cloud (EWC) with key ECMWF technologies from the Destination Earth project. This blueprint serves as an example of how Member States can effectively leverage EWC resources while streamlining workflows through automation and real-time data processing.

The workflow follows an automated sequence: IFS forecasts are pushed to the European Weather Cloud, where Aviso, continuously polling the storage bucket, triggers a preprocessing job containerized with Earthkit-Data. This step is followed by running the Flexpart model in a container, and finally, a plotting application generates visual outputs. The architecture of this workflow is illustrated in the diagram below.

Workflow architecture:

Operational since September 2023, the EWC is still evolving. This use case evaluated EWC’s current capabilities, identifying missing components needed for automated workflows and providing feedback to the EWC team. The key priorities include ensuring that the workflow, which runs regularly for short periods, takes full advantage of cloud-based deployment by creating resources on demand following the pay-as-you-go principle. Additionally, fostering collaboration through a shared repository allows users to store, access, and run container images or applications seamlessly, enhancing efficiency across the platform.

To refine and validate our approach, we collaborated with key partners actively working with Flexpart on the EWC and Atos. The Royal Meteorological Institute (RMI) Belgium, which operates a web application for Flexpart simulations on the EWC, had encountered challenges with data retrieval. They expressed strong interest in our approach using Polytope to optimize data access.

Additionally, we engaged with partners of the University of Vienna, the main developers of Flexpart, to exchange insights on our activities on the European Weather Cloud. They shared similar limitations, particularly in workflow automation, data retrieval, and infrastructure scalability. They were also interested in deploying a service that runs Flexpart on Kubernetes within the EWC but faced the same constraints. These limitations, further detailed in the Key Outcomes and Identified Limitations section below, underscore the need for improvements in resource management and infrastructure flexibility within the EWC.

Key Outcomes and Identified Limitations

• Cloud Resource Management: The workflow currently runs on a continuously active virtual machine instead of dynamically provisioning resources (e.g., via Kubernetes). This is inefficient, but EWC is expected to support short-lived job execution by June 2025.

• Data Retrieval via Polytope: Some IFS fields required for Flexpart are currently unavailable on Polytope ECMWF. The Flexpart community has expressed strong interest in accessing these fields via Polytope instead of the MARS client. Since integrating Polytope was a key objective of the pilot project, enabling this retrieval remains a priority. To address this, ECMWF are working on a Data Bridge, hosted on SwissTwin at the Swiss National Supercomputing Centre (CSCS), which will facilitate access to this data for all ECMWF Member States. Estimated timeline: End of Q2 2025.

• Infrastructure as Code (IaC): Initially, deployment required manual setup, which was time-consuming and inconsistent. As of mid-March 2025, Terraform-based IaC is available, improving reproducibility and scalability.

• Container & IaC Repository: No shared repository exists within EWC for container images and IaC configurations. While DockerHub is an alternative, a dedicated EWC registry would enhance collaboration.

Remaining Actions Within the Project Timeline Through End of 2025

▢ Data Retrieval via Polytope: Once the required input data becomes available on Polytope, integrate it into the workflow for seamless data retrieval.

▢ Transition to IaC-Based Deployment: Replace the static virtual machine with a fully automated, on-demand infrastructure setup.

▢ Establish a Shared Repository: Advocate for a centralized container and IaC registry within EWC to streamline deployment and collaboration.

Modernizing Flexpart Preprocessing with Earthkit-Data

Resulting from our discussions with colleagues at the University of Vienna, the main developers of Flexpart, we explored modernizing parts of the existing flex_extract software, including its Fortran-based preprocessing component, with a Python-based solution using Earthkit-Data for seamless handling of GRIB data.

The preprocessing step is essential for preparing ECMWF meteorological fields as input for Flexpart. It involves performing various operations on the input data, such as deaggregating precipitation values and other transformations required for accurate atmospheric transport modeling. With Earthkit-Data, these fields can be efficiently read in GRIB format and converted into Xarray datasets, enabling powerful multi-dimensional operations that enhance readability, flexibility, and efficiency.

Key Outcomes

• Improved Readability & Maintainability: Transitioning from Fortran to Python enhances code clarity and flexibility.
• Simplified GRIB Handling: Earthkit-Data simplifies ECMWF data processing

Remaining Actions within the Project Timeline Through End of 2025

▢ Collaborate with University of Vienna for refinements

▢ Publish the processing with earthkit-data as Flexpart tool

Deploying Polytope and FDB at MeteoSwiss

MeteoSwiss has actively contributed to the deployment of Polytope and FDB, aligning its infrastructure with ECMWF’s evolving data methodologies to enhance model output accessibility for public users and downstream applications.

To support this effort, bi-weekly meetings have been held for over a year, facilitating regular exchanges on the deployment of Polytope and FDB at MeteoSwiss. The primary objective is to improve ICON model output accessibility for downstream applications, ensuring seamless data retrieval and integration.

As part of this collaboration, we actively contribute to ECMWF developments by reporting bugs, providing feedback, and maintaining continuous engagement through meetings and shared goals. Our contributions focus on the following key areas:

• Deployment Consolidation: Streamlining the release deployment process for Polytope and FDB from ECMWF.
• Extended Grid Support: Implementing ICON grid and rotated latitude/longitude support in Polytope.
• Server Robustness & Performance Evaluation: Developing a framework to assess performance, including stress tests for multiple users and parallel large data requests, to define Polytope’s scalability limits—an essential goal that we have successfully achieved.

Workflow Architecture:

Remaining Actions within the Project Timeline Through End of 2025

▢ Monitoring & Reliability – Implement monitoring for stable operation, centralized logging, automated error reporting, and performance tracking with request tracing, health metrics, and alerts.

▢ Deployment Improvements – Streamline image building and deployment, enable rollback capabilities, maintain a stable production version while testing new releases (DEPL), automate container build chains, and introduce alerts for failed deployments.

Facilitating Knowledge Sharing through Webinars

Remaining Actions Within the Project Timeline Through End of 2025

▢ Webinar: Cloud-Based Technologies in Member States. The next webinar will explore how Member States leverage cloud-based technologies for weather and climate applications. It will highlight real-world use cases, challenges, and best practices, fostering collaboration and knowledge exchange on cloud adoption in meteorology.