You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Adaptation to emerging technologies pilot project  

Summary Report 2026 

August 2026 

  

Summary 

This report provides an update on the progress of the Adaptation to Emerging Technologies pilot project led by MeteoSwiss. It covers activities undertaken since the previous reporting period in Q3 2025, including developments completed up to August 2026, with a focus on consolidating the project’s technical developments into reusable examples, blueprints and contributions that can support the wider uptake of ECMWF technologies by Member States. 

  

  1. Introduction & Background 

The Adaptation to Emerging Technologies pilot project was established to explore how ECMWF and its Member and Co-operating States can make use of new technologies for accessing and processing Numerical Weather Prediction (NWP) data. As data volumes continue to grow, traditional approaches based on transferring large datasets before processing them are becoming increasingly difficult to scale. The project therefore investigates new approaches in which users can access, select, and process data closer to where it is produced and stored. 

To support this, the pilot has worked with technologies developed by ECMWF, including FDB, Polytope, earthkit and Aviso, as well as resources provided through the European Weather Cloud (EWC). Rather than focusing only on the technologies themselves, the project develops practical workflows and open-source blueprints to demonstrate how they can be combined and adapted for real applications. The aim is to provide experience, examples, and feedback that can benefit both ECMWF and the wider Member State community. 

During its first phase (2024/25), the project developed and tested a number of such workflows. For the one-year extension approved in 2025, the emphasis has shifted towards consolidating these results, making them easier to reuse and share, and contributing improvements back to the underlying ECMWF technologies and software ecosystems. Further information, progress reports, presentations and project outputs are available on the project’s Confluence space. 

The main objectives of the project are to: 

  • O1 – Develop and demonstrate reusable workflows and blueprints for efficient access to and processing of NWP data using emerging ECMWF technologies. 
  • O2 – Support the integration of these technologies with the European Weather Cloud and Member State environments, enabling data to be processed or reduced before transferring where appropriate. 
  • O3 – Strengthen collaboration and knowledge exchange across ECMWF and its Member States, using practical use cases to identify common requirements and provide feedback on emerging technologies. 
  • O4 – Support the longer-term uptake and sustainability of successful approaches, including through open-source developments, documentation, and community dissemination. 

For the extension year, the work is organised around three technical work streams defined in the Statement of Work for the project extension: 

  • WS1 – earthkit–Anemoi alignment: assess and reduce duplicated data-handling functionality and support the progressive use of earthkit as a common data layer in Anemoi. 
  • WS2 – FLEXPART on the European Weather Cloud: consolidate the FLEXPART-IFS workflow into a documented, deployable blueprint for reuse through the EWC Community Hub.  
  • WS3 – Native unstructured-grid support: validate and extend earthkit and Polytope capabilities for loading, retrieving, regridding and processing native ICON data, supported by reusable example notebooks. 

 

  1. Personnel involved 
  • MeteoSwiss:  
  • Nina Burgdorfer (1 FTE for technical development is supported with project funding), Simon Bolt, Victoria Cherkas, Katrin Ehlert, Christian Kanesan, Lasse Meinen, Noemi Nellen and Carlos Osuna. 
  • ECMWF: 
  • Polytope/FDB: Tiago Quintino, James Hawkes, Emanuele Danovaro, Peter Tsrunchev, Mathilde Leuridan and Christopher Bradley 
  • earthkit: Sandor Kertesz, Pedro Maciel, Iain Russel and James Varndell 

 

  1. Pilot project activities, achievements and outcomes 

Building on the original project objectives, activities during the extension year have focused on consolidating the results of the first project phase into reusable workflows, shared software capabilities and practical examples that can support wider uptake of ECMWF technologies. 

 

3.1 Summary of the action 

 

Table 1: Summary of progress against the extension-year work streams and planned outputs, mapped to the project objectives, as of August 2026. Each topic is described in more detail below, with links to relevant public outputs and examples where available. 

Extension work stream and planned output 

Objectives addressed 

Progress and achievements to date (August 2026)   

WS1 – Integration of earthkit in the Anemoi framework (D1.1 Migration to earthkit; D1.2 Missing or improved earthkit functionality) 

 

O3 · O4 

 

The project established an exchange with the Anemoi machine-learning community, with the goal to assess overlaps between Anemoi and earthkit in the handling of meteorological data. Some functionality currently implemented or wrapped within Anemoi can increasingly be provided by earthkit as a common data-handling layer, allowing Anemoi to focus more clearly on its machine-learning workflows. Following the stable earthkit 1.0 release at the end of June 2026, existing migration work within Anemoi is updating the packages to the new interfaces and reducing duplicated data-handling code. Testing of the migrated interfaces and identification of remaining gaps continue during Q3. 

WS2 – Publishing the FLEXPART workflow as an EWC blueprint 

(D2.1 Deployable blueprint; D2.2 Documentation and user guidance) 

O1 · O2 · O4 

The FLEXPART atmospheric dispersion model has been integrated into a documented, deployable blueprint, and published on the EWC Community Hub, where EWC users can discover and reuse applications. The blueprint automates the full FLEXPART chain, from retrieval and preprocessing of ECMWF forecast data to model execution and visualisation, and can be deployed using EWCCLI, the EWC command-line tool. Beyond the FLEXPART use case, it provides a practical example of how Member States can turn technical workflows into reproducible, shareable EWC resources. 

WS3 – Support for native unstructured-grid data and meteorological processing in the earthkit ecosystem (D3.1 Testing, gaps and feedback; D3.2 Demonstration notebooks) 

O1 · O2 · O3 

The project has advanced and demonstrated support for unstructured-grid data across ECMWF’s data-access and processing tools. Using the operational ICON model configurations at MeteoSwiss as practical test cases, notebooks demonstrate how earthkit can load, regrid and visualise these data, while Polytope enables selective retrieval of areas, time series, trajectories and profiles. The resulting reusable workflows and examples demonstrate how ECMWF tools can be applied to native unstructured-grid data and provide a basis for evaluating similar approaches for other Member State datasets. Further work is ongoing during Q3 and Q4 to consolidate these workflows and extend meteorological processing capabilities in earthkit-meteo. 

Cross-cutting –  Community dissemination and reuse 

O3 · O4 

The extension year has increased the emphasis on making project outputs openly accessible and reusable beyond their original implementation. Public notebooks demonstrate complementary ways of working with NWP data: open-data access and processing with earthkit, and data access and processing with FDB, Polytope and earthkit. Together, the examples show how NWP data can be retrieved, processed and visualised using the technologies explored by the project. The EWC blueprint is likewise accompanied by source code, deployment documentation and usage guidance. The experience of moving from a technical prototype to a reusable Community Hub resource will also be presented at the EWC User Workshop on 22 September 2026, providing a practical example for other users interested in publishing and sharing their own EWC workflows. 

 

3.2 Results and activities 

 

WS1 – Alignment of earthkit and Anemoi data-processing components 

Anemoi, ECMWF’s machine-learning framework for developing and running data-driven weather models, and earthkit evolved partly in parallel, leading to some overlapping and duplicated meteorological data-handling functionality. Maintaining similar capabilities in both ecosystems increases complexity and maintenance effort. The aim of this workstream is therefore to use earthkit as the common data-handling layer, allowing Anemoi to focus on its machine-learning-specific functionality. 

The project reviewed the relevant Anemoi components and identified the clearest duplication in anemoi-datasets, where custom xarray handling covered conversion to field objects, coordinate and metadata handling, time handling and variable selection. These functions can now increasingly rely on the common earthkit Field/FieldList abstractions instead of being maintained separately.  

Following the stable earthkit 1.0 release, migration work is now updating anemoi-datasets, anemoi-transform and anemoi-inference to the new interfaces and reducing duplicated data-handling code.  

The next step is to test the migrated interfaces with representative Anemoi workflows, gather community feedback and identify any remaining gaps that should be addressed in earthkit. 

WS2 – Reusable FLEXPART blueprint on the European Weather Cloud 

The FLEXPART-IFS workflow developed during the first phase of the project has been consolidated into a reusable blueprint and published on the EWC Community Hub. The Community Hub provides a shared catalogue where the European meteorological community can discover and reuse applications, deployment templates and other cloud resources. 

 

 

Figure 1: Deployment and execution of the FLEXPART-IFS workflow on the European Weather Cloud 

As illustrated in Figure 1, the blueprint reduces deployment of the complete workflow to only a few user commands. From a local computer, a user with access to an EWC tenancy can deploy a pre-configured virtual machine and then start a dispersion simulation by selecting, for example, the forecast cycle, simulation length and release location. The workflow then automatically retrieves the required IFS forecast data through ECMWF MARS, preprocesses it using flexprep, developed within the project as an earthkit-based reimplementation of the original FLEXPART preprocessing, runs FLEXPART-IFS in a container and generates ready-to-view PDF plots. 

The deployment itself is implemented as an Ansible blueprint, bringing the individual software components and their configuration together into a reproducible EWC environment. Users therefore do not need to install and integrate each component independently. They require an EWC tenancy and their own ECMWF credentials and data entitlements for retrieving the required IFS forecasts. 

Beyond the FLEXPART application, the main value of this work is the reusable deployment pattern: an application developed by a Member State can be packaged, documented and published through the Community Hub so that other users can deploy it within their own EWC environment. The FLEXPART blueprint therefore provides a practical example for Member States interested in making their own workflows easier to distribute and reuse. 

The experience of moving from a technical prototype to a published Community Hub item will also be presented at the EWC User Workshop on 22 September 2026, providing a practical example for users interested in publishing their own workflows. 

EWC Community Hub 
 FLEXPART-IFS Atmospheric Dispersion Workflow 
 View the published blueprint → 

 

WS3 – Reusable data access and processing for unstructured grids 

During the extension year, the project has enabled native unstructured-grid data to be used across key parts of the earthkit workflow. Using operational ICON data from MeteoSwiss as a practical test case, earthkit can now load native-grid fields together with the latitude and longitude of each grid point, convert them into the widely used xarray format, regrid them to regular latitude–longitude grids and visualise the results. This closes an important gap that previously limited the use of unstructured-grid data with common earthkit processing tools. 

 

The work was developed jointly with ECMWF through biweekly technical exchanges, using operational MeteoSwiss data for testing and feedback. The resulting workflow has been consolidated into public example notebooks, demonstrating the sequence from native-grid loading and xarray conversion to regridding and plotting. 

 

On the data-access side, the project also worked with Polytope, ECMWF’s service for extracting selected subsets of forecast data. Polytope’s feature-extraction algorithms can now operate directly on the native ICON unstructured grid, which was not previously possible. This allows users to select only the data required for geographical areas, time series, trajectories or vertical profiles, without first regridding the data. Together with the earthkit developments, this enables selective access to and processing of data while retaining the native unstructured grid. 

 

Public example notebooks 
earthkit workflow for unstructured-grid data 

View the earthkit notebook → 
 Polytope feature extraction on native unstructured-grid data 
 View the Polytope notebooks → 

 

Further work will focus on consolidating these workflows and extending meteorological processing capabilities in earthkit-meteo. 

 

3.3 Challenges 

A key challenge is involving more Member States in concrete use cases. The project continues to seek wider engagement through community events and has also been raised in discussions at ECMWF TAC/PAC meetings, but additional Member State use cases remain limited. Broader participation would help assess how the technologies explored in the project can be applied to different Member State use cases and operational environments. 

The later-than-planned release of earthkit 1.0, originally expected by the end of Q1 2026, has also shifted parts of the earthkit–Anemoi alignment and testing into Q3. This reduces the time available for testing the migrated interfaces and consolidating the remaining work during the extension year. 

3.4 Presentations and Publications 

  • 2025.12.02, Machine Learning Pilot Project All-Hands presentation, Reading UK 
  • 2026.05.04 EGU Conference Workflow approaches session, Vienna AUT 
  •  2026.07.06 Machine Learning Pilot Project All-Hands, Wildhaus CH 

 

  • No labels