Statement of Work and administrative procedures

for the extension of the

ECMWF and MeteoSwiss collaboration on the pilot project “Adaptation to Emerging Technologies”

Executive Summary

The ECMWF Pilot Project on “Adaptation to Emerging Technologies” was initiated by the Council in December 2022 (see ECMWF/C/105(22)15 and ECMWF/C/105(22)D) with the goal to build a European user community around emerging technologies at ECMWF for data accessibility and data handling, namely FDB, Polytope, Earthkit. To this end, the pilot project established blueprints and use cases to showcase new workflows within its first two years of implementation.

Based on the successful implementation of the projects goals, the high benefit of additional time to consolidate the blueprints and the value of outreach to share the achievements with the European community, Council decided in July 2025 (ECMWF/C/110(25)9 Rev.1 and ECMWF/C/110(25)D) to extend the project for 1 year at a cost of £100K for ECMWF. MeteoSwiss continues to offer in kind contributions of at least 1 FTE for the areas covered by the project including data harmonization and interoperability, Gridefix, evaluations for various access patterns and use cases, and port of ICON to gt4py.

Objectives

Main objectives:

• foster collaboration with member states (MS) in the area of emerging technologies for data production, access and distributed data proximate compute for NWP, bring together expertise, build a blueprint workflow relevant to several member states involving ECMWF’s atos and the EWC resources, but also consider deployment within MSs or public cloud services.
• Setting up a regular dialogue on GPU or other accelerators within a discussion forum on (merging) adaptation strategies such as DSLs would be the other topic.
• Exploring funding opportunities to sustain these activities in the long term.

Work Streams and Deliverables

The project focuses on developing and validating technologies for data pipelines and end-to-end workflows that enable Member States to integrate data processing directly into their operational environments. Services such as FDB, Earthkit-data, Polytope and Aviso are being made accessible from EWC resources, supporting workflows in which data can be processed, transformed or reduced before being transferred to Member State systems.

The project aims to provide a set of blueprints for deploying such pipelines. These blueprints offer ready-to-use patterns that Member States can adapt to their own applications, and — with the introduction of the EWC Marketplace — can now be shared more widely within the community to support adoption and reuse. The work centres on the following key technologies:

• Aviso – event-notification system used to trigger workflow components as new data becomes available.
• FDB – field database enabling structured access to IFS production data.
• Earthkit-data – Python framework for managing GRIB data and building NumPy/xarray-based processing applications.
• Polytope – service for efficient extraction of selected multidimensional subsets from FDB.

Work Stream 1: Integration of Earthkit-Data in the Anemoi Framework

As part of the extended collaboration with Member States, this workstream focuses on strengthening the integration of Earthkit components within the Anemoi data-processing ecosystem. While anemoi-datasets already relies on earthkit-data in several places, many functions are wrapped, filtered, or reimplemented, and other parts of the framework still use custom code where existing Earthkit functionality—either in earthkit-data or earthkit-meteo—could be used directly. Similar patterns appear in anemoi-transforms, where some transformations could also be based on Earthkit’s meteorological tooling.

The objectives for the extension year are:

• to identify and document where anemoi-datasets and anemoi-transforms wrap or duplicate functionality already available in earthkit-data or earthkit-meteo;
• to assess where Earthkit components can replace custom implementations;
• to contribute missing or enhanced features to Earthkit so that it fully supports Anemoi’s data-handling and transformation requirements.

Anemoi intends to adopt Earthkit more broadly once earthkit-data reaches a stable 1.0 release (expected by end of Q1 2026). This workstream ensures that such a transition is feasible, that required functionality is in place, and that unnecessary wrappers can be removed progressively.

For the project, this activity strengthens the uptake of ECMWF’s data-access technologies across the Member State machine-learning community contributing to Anemoi. Adopting Earthkit as the common data layer reduces fragmentation in data-handling practices and supports a more coherent ecosystem. For Anemoi, the longer-term benefit is a leaner, more maintainable code base, allowing greater focus on dataset preparation for machine-learning workflows rather than on low-level data plumbing.

Deliverables

Work Stream 2: Publishing the Flexpart Dispersion Workflow as an EWC Blueprint

The Flexpart dispersion workflow developed in the previous phase provides a complete chain from retrieving IFS inputs, pre-processing them with Earthkit, running the Flexpart model, and generating dispersion plots. With the workflow now implemented, the focus for this final year is to prepare it for reuse by the wider EWC community.

The aim is to package the workflow as a blueprint that can be easily retrieved, deployed, and adapted by users, if possible, through the EWC Marketplace (https://europeanweather.cloud/community-hub). This will allow Member States and other users to run the full workflow within their own EWC projects without redeveloping its individual components.

Objectives

• Prepare the existing workflow components (data access, preprocessing, model execution, plotting) for publication as a reusable blueprint.
• Ensure the workflow is self-contained, documented, and configurable for general use beyond its original setup.
• Publish the blueprint on the EWC Marketplace and provide basic user guidance.

Deliverables

Work Stream 3: Support for ICON Native Grid and Meteorological Processing in the Earthkit Ecosystem

Following the joint hackathon with ECMWF in October 2025, this workstream focuses on supporting the ICON native unstructured grid and integrating additional meteorological processing capabilities into the Earthkit ecosystem. With the support of the Earthkit development team, we ensure the entire Earthkit ecosystem supports the native ICON data, model and postprocessing. The work aims at providing a working prototype which will allow Member States to use and develop post-processing data pipelines for ICON data based on the Earthkit framework. The workstream addresses three areas:

1. Loading, processing, regridding and plotting of ICON data within Earthkit
   1. Support for ICON native grid data in earthkit-data, ensuring fields can be loaded and converted to Xarray with the lat/lon coordinates of the unstructured grid cells.
   2. Validation of ICON interoperability across the entire ecosystem (earthkit-data → earthkit-meteo → earthkit-regrid → earthkit-plots).
   3. Coordination and feedback on regridding between ICON and regular grids.
2. Retrieval of ICON data through Polytope
   1. Coordination and testing of Polytope’s extraction capabilities for ICON native grid fields.
   2. Feedback to ensure feature selection works reliably for unstructured grid data.
3. Meteorological processing capabilities in Earthkit
   1. Contribution to the design and testing of vertical-coordinate transformations in earthkit-meteo (terrain-following coordinates, pressure levels, height AMSL, theta levels).
   2. Validation and feedback on new diagnostic computations such as CAPE, potential vorticity and others required by existing workflows.

Deliverables

Project Management

The internal developments are integrated within a team of developers at MeteoSwiss working on deployments of FDB and data processing frameworks using earthkit.

We will establish regular coordination meetings with the ECMWF and Destination Earth developers. This will ensure that the blueprints and data processing frameworks are developed with usage patterns well aligned with the design of the technology and infrastructure at EWC.

The project will be managed in an agile way with specific objectives for each cycle.

The objective of the current and past cycles will be documented in a confluence space at ECMWF so that all stakeholders and member state can follow the progress.

Regular (frequency to be decided) meetings with the main stakeholder will take place to discuss and plan the objectives of future development cycles.

Following agile methodologies, project plan and goals can be modified during the project in order to improve the original proposal of the architecture and the proposed work in the light of results obtained during the project. Decisions to adapt timelines and objectives will be discussed and agreed with stakeholders and properly disseminated.

The project will adopt open development practices, preferably using Apache 2.0 licences for software packages and CC-BY-4 for datasets. All code developed will be open source from the beginning. Technical planning and development of the work will also be open. The exception is the use of component or meteorological applications that are already subject to a proprietary license. 

Dissemination

In addition to the planning and documentation of work we will document progress and results in the confluence of ECMWF.

Furthermore, we will organize regular (tentatively every six months) open review meetings where we will provide a status update, open to all member states interested.

A periodic report summarizing the main results will be distributed at least 2 weeks before the meeting. Reports should also includes latests results on explorations of GPUs for data processing and the use of DSLs (developed through other projects like EXCLAIM).

All code will be developed as open source. A final released of the blueprints will be disseminated with user documentation.

Timeline

• M1 — January 2026
  Publication on the EWC Marketplace of the Flexpart dispersion workflow as a reusable blueprint, including accompanying documentation.
• M2 — End of Q1 2026
  Interim report covering:
  • tests and feedback for ICON native-grid support (Earthkit, Polytope, regridding, plots),
  • initial list of functions in anemoi-datasets and anemoi-transforms that duplicate or could be transferred to Earthkit, together with coordination outcomes with ECMWF on the planned scope of Earthkit 1.0 and what can realistically be integrated there.
• M3 — Mid-2026
  Delivery of example notebooks demonstrating data retrieval and processing with Earthkit, including loading ICON data, regridding, diagnostic computations, and workflow examples for the community.
• M4 — End of Q3 2026
  Second interim report summarising the results of further testing, remaining gaps, and feedback on Earthkit developments (ICON support, meteorological operators, vertical coordinate handling).
• M5 — End of 2026
  Final report and consolidation of all workstreams, including:
  • updated blueprints and documentation,
  • final Anemoi–Earthkit assessment and recommendations,
  • final validation notebooks,
  • dissemination of all outputs through the EWC Marketplace and project confluence space.

Payment Schedule

50 % to be paid after the submission of the Q1 2026 report and 50 % to be paid after submission of the Q3/2026 (final) report.

The exchange rate will be decided at the invoice time.

Key Personnel

Project Lead MeteoSwiss – Carlos Osuna (co-lead of Computing in Numerical Development)

Project Lead ECMWF – Nils Wedi (Digital Technology Lead for Destination Earth)

Technical Lead ECMWF – Tiago Quintino

Technical Lead MeteoSwiss – Victoria Cherkas

Technical Lead UK Met Office – Richard Lawrence

1 FTE (1 year) granted by this pilot project for all development activities and dissemination/documentation tasks.

In-kind resources from MeteoSwiss:

• Victoria Cherkas (software engineer) will contribute to the setups requiring FDB/pyFDB and the development of the python data processing framework using earthkit. Also the setup of aviso configurations in EWC.
• Christian Kanesan (software engineer) will contribute to the development of the data processing framework using earthkit.
• Petra Baumann (software and data engineer) will contribute to data harmonization and grib data related tasks.
• Carlos Osuna and Katrin Ehlert will contribute to project management and dissemination