ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

Table of contents

Highlights

Software stack ready for next operational IFS cycle

With the upcoming Implementation of IFS Cycle 47r3, this set of defaults is ready to handle the data produced, including all the new parameters introduced in this cycle.

ECMWF software improvements

See the blog posts for the latest two software releases to learn about what is new in some of our own software packages and libraries such as ecCodes, Magics or Metview.

• May 2021 software release
• August 2021 software release

All the new versions come with a number of bug fixes and improvements. You may see the changelogs for any individual package or library in the list below if you want to have all the details.

How to use the new versions for testing BEFORE the update

The new versions of the software packages can be selected by using the oct21 tag in modules. For example:

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

Change list

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

Table of contents

Highlights

Software stack ready for next operational IFS cycle

With the upcoming Implementation of IFS Cycle 47r2, this set of defaults is ready to handle the data produced, including all the new parameters introduced in this cycle and the update to Tropical Cyclone tracks.

Python 3.8 becomes default

For our Python users, we offer now a more recent version of Python 3.8 with the usual extra modules found on the previous default 3.6 which would be out of official maintenance at the end of the year.

For those still using Python 2, note that this major version of Python reached its end of life on 1st January 2020, If you haven't done moved to Python 3.6 yet, please see Migration to Python 3 for some guidance on where to start. We are providing the latest Python 2.7 for compatibility purposes only.

ECMWF software improvements

See the blog posts for the latest two software releases to learn about what is new in some of our own software packages and libraries such as ecCodes, Magics or Metview.

• March 2021 software release
• January 2021 software release

Potentially disruptive changes in ecCodes

It is worth highligting a number of changes in ecCodes that could have an impact in specific user workflows:

• The GRIB2 representation of parameter 238 (Temperature of snow layer - tsn) has been modified. See the new encoding here: https://apps-test.ecmwf.int/codes/grib/param-db/?id=238
• The command codes_info -v no longer outputs a trailing space. Some user scripts may need to be modified.
• grib_get_data output is slightly modified: Words on the first line are now separated by spaces not commas.
• The tools now check that the output file is not the same as the input to avoid overwriting/corrupting the source.
• The MARS keys levtype and levelist have been removed from the GRIB2 templates 4.53 and 4.54.

A lot of other bugfixes and improvements

All the new versions, both ECMWF and 3rd Party, come with a number of bug fixes and improvements. You may see the Change logs for any individual package or library in the list below if you want to have all the details.

How to use the new versions for testing BEFORE the update

The new versions of the software packages can be selected by using the may21 tag in modules. For example:

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

Change list

ECMWF Software

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

Third-party software

Some of the versions in this list may already be the default for a specific package on a certain platform, so no changes will be made in those cases. After the update, all platforms will have the same defaults for all these packages.

ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

Table of contents

Highlights

A number of bug fixes and performance improvements have been introduced on the different versions to become default. Here are some of the most relevant new features in respect to the previous default set of versions:

• ecCodes, the heart of GRIB and BUFR handling, brings performance improvements, new parameters, contributions from users and several smaller features/bug fixes including support for the Mercator projection. Please note that ecCodes' Python 2 interface is no longer being developed. Users are strongly advised to migrate to Python 3.
• Magics, ECMWF's plotting library, adds some smaller fixes and additional improvements including :
  • combined with the latest ecCodes, Magics brings plotting support, including interactive Cursor Data, for GRIBs encoded with various projections such as Mercator and Lambert variants.
               
  • a major code clean-up has reduced the number of dependencies. The Boost library and Python are not required to install Magics anymore.
  • fix issue with logging that was degrading observation plotting speed when used from Python
  • fix issue in python when passing single element array.
• Metview, ECMWF's interactive and batch software for manipulation and plotting of data, brings several new features and functions, such as:
  • much improved speed of plotting in-memory arrays of data using the Input Visualiser thanks to allowing vectors (Macro language) or numpy arrays (Python) to be used as data
  • the last two modules that were originally implemented in Fortran (Spectra and Potential Temperature) have been converted to C++, meaning they will be available to more users
  • fixes for macOS
  • new Gallery examples
  • a new regridding module that exposes many new interpolation methods for expert users
  • user interface improvements such as a new preview panel that allows quick inspection of icons
  • the long-awaited ability to filter for parameters in the icon editors makes its way into this release
  • a whole raft of new Python/Macro functions for thermodynamic computations and plotting utilities
  • more - see the release notes!

    

    Metview's new Regrid module brings new possibilities in gridding and transformation of data

• CodesUI, the standalone combination of Metview's GRIB and BUFR examiners, added syntax highlighting for textual GRIB dumps and fixed an issue where the BUFR filter could unexpectedly crash.

How to use the new versions for testing BEFORE the update

The new versions of the software packages can be selected by using the nov20 tag in modules. For example:

module swap metview/nov20
module load Magics/nov20
module load bufrdc/nov20
module swap eccodes/nov20 

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

module swap metview/old
module load Magics/old
module load bufrdc/old
module swap eccodes/old 

Version changes

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

Caveats

ecCodes 2.19 and Snow Depth Parameter (141)

ParameterNumber for Snow Depth (141) is now 254 instead of 11 when encoding in GRIB2 format.

The GRIB edition 2 for 141 encoding was wrong in previous versions of ecCodes: The parameterNumber should be 60, not 11 because 11 is the physical depth not the one converted to water volume.

In older versions, both paramIds 141 and 3066 shared the same GRIB2 representation and the consequence of that was 141 was obfuscated by 3066 (water converted depth is obfuscated but the true physical depth!)

From ecCodes 2.19 onwards, this parameter uses local GRIB2 encoding in ECMWF local table 1:

discipline = 0;
parameterCategory = 1;
parameterNumber = 254;
localTablesVersion = 1 ;

ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

Table of contents

Highlights

Software stack ready for next operational IFS cycle

With the upcoming Implementation of IFS Cycle 47r1, this set of defaults is ready to handle the data produced, including all the new parameters introduced in this cycle.

No Python 2 upgrade

As Python 2 reached its end of life on 1st January 2020, no newer versions of the Python 2 installations and their associated subpackages such as numpy and others. Older versions remain available for those who haven't migrated to Python 3. If you haven't done so yet, please see Migration to Python 3 for some guidance on where to start.

Metview improvements

Metview, ECMWF's interactive and batch software for manipulation and plotting of data, brings a number of new features and improvements, such as:

• easier access to the layer metadata in the interactive plot window
• a new set of pre-defined areas in the Geographical View icon
• improvements to the GRIB and BUFR examiners (and hence also in CodesUI)
• improvements to the existing thermodynamic functions
• plus a new function to compute relative humidity from temperature and dewpoint, a new way to control the font sizes in the user interface

Codes UI efficiency improvements

CodesUI, the standalone combination of Metview's GRIB and BUFR examiners, provides a highly improved initial BUFR message scan (about 50-100 times faster than before), a filter bar for the GRIB namespace list and extra information about BUFR elements.

A lot of other bugfixes and improvements

All the new versions, both ECMWF and 3rd Party, come with a number of bug fixes and improvements. You may see the Change logs for any individual package or library in the list below if you want to have all the details.

How to use the new versions for testing BEFORE the update

The new versions of the software packages can be selected by using the jun20 tag in modules. For example:

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

Change list

ECMWF Software

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

Third-party software

Some of the versions in this list may already be the default for a specific package on a certain platform, so no changes will be made in those cases. After the update, all platforms will have the same defaults for all these packages.

As previously announced, the Product Requirements Editor will replace the existing requirement-editing interface, used by ECMWF Member and Cooperating Sates for the management of real-time dissemination requirements, on Tuesday 18^th February 2020.

For full details and final testing, please refer to the following blog post:  Introducing the Product Requirements Editor

ECMWF are pleased to announce the availability of a beta version of a new real-time requirement-editing interface.

Who is it for?

The Product Requirements Editor is an intuitive platform for the editing of real-time data requirements.

User group	Access to the legacy platform	Access to beta phase of the new platform	Access to the new platform (18th February 2020)
Member & Cooperating State		- 17th January - 18th February
NMHS licence holders	(access to requirements was only available via ECPDS)
Maximum charge customers	(access to requirements was only available via ECPDS)

Highlights & new features

The Product Requirements Editor offers a variety of new features designed to assist users in the management of real-time data delivery.

Examples features include:

How to test

To ensure that you are ready to use the new application from Tuesday 18^th February 2020 please test at your earliest convenience.

Further updates

Become a watcher of this blog if you would like to receive further updates regarding the Product Requirements Editor, including announcements of Webinars.

GRIBEX, our old GRIB edition 1 decoder/encoder, has been discontinued for years, as it was replaced by GRIB API and then superseded by ecCodes. However, we are aware that it may still be used by some of our users. As part of the retirement of EMOSLIB process, access to the current location of the GRIBEX tables is to be closed. When this happens, any existing program or tool using EMOSLIB for the GRIB decoding/encoding will stop working. This should raise awareness of the places in your workflow that need your attention before our migration to the new data centre.

When?

The session is scheduled for Wednesday 29 January 2020 at 10:45 UTC

Where?

The platforms affected are:

• All clusters in our High Performance Computing Facility, mainly CCA and CCB
• ECGATE
• Linux Clusters
• Linux Desktops

Actions to take

If you are currently using an old version of emoslib for the GRIB decoding, you will need to migrate those to ecCodes as soon as possible. As a starting point please check here.

However, to temporarily continue working with the existing executables, libraries and tools as they are, you may load the emos_gribex_tables module before running them. This will make the GRIBEX tables available.

module load emos_gribex_tables

Throughout the years, new software packages and multiple versions of them have been deployed on our different computing platforms. In preparation for our future migration to Bologna - New Data Centre, we will be streamlining the number of software packages and their versions available, so the majority of the user base is on recent software stacks prior to our move.

In a first stage in September, we removed the modules for those versions. Now is time to close the access to the actual deployments and installations for those software packages and libraries. Although they are not going to be deleted at this point, they will become unavailable.

If you have any concerns, please get in touch with our servicedesk@ecmwf.int.

When?

The session is scheduled for Wednesday 27 November 2019 at 11:00 UTC

Where?

The platforms affected are:

• All clusters in our High Performance Computing Facility, mainly CCA and CCB
• ECGATE
• Linux Clusters
• Linux Desktops

Packages to be retired

ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

Table of contents

Highlights

A number of bug fixes and performance improvements have been introduced on the different versions to become default. Here are some of the most relevant new features:

• ecCodes, the heart of GRIB and BUFR handling, brings its usual support for new GRIB parameters along with numerous fixes for extra stability. There are performance improvements for decoding/encoding GRIB data in a multi-threaded program as well as new options for the grib_ls and grib_dump tools.
• Metview, ECMWF's interactive and batch software for manipulation and plotting of data, brings several new Macro/Python functions, including ml_to_hl(), which converts GRIB data from model levels to height above ground or sea level. Also in this release is an improved inline Help sidebar in the Code Editor.
• CodesUI, the standalone combination of Metview's GRIB and BUFR examiners, added new tab called Descriptors to show all the expanded descriptors (metadata only) of the given message.
• Magics, ECMWF's plotting library, adds some smaller fixes.

How to use the new versions for testing BEFORE the update

The new versions of the software packages can be selected by using the nov19 tag in modules. For example:

module swap metview/nov19
module load Magics/nov19
module load bufrdc/nov19
module swap eccodes/nov19 

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

module swap metview/old
module load Magics/old
module load bufrdc/old
module swap eccodes/old 

Version changes

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

Python is a very popular tool widely used at ECMWF and elsewhere. If you do not use it, you can ignore this message.

As you may know, the official support for Python 2 ends this year. Since Python 3 is not backwards-compatible, steps must be taken to migrate to this new major version. With the lack of official support, we may not be able to install Python 2 on future platforms. Keeping it up-to-date on the existing is also proving increasingly difficult, since many extra modules have dropped their support for Python 2 already. Note that this is not something affecting ECMWF only. If you run Python workloads elsewhere, you will face the same issues.

At ECMWF we have been preparing in order to ensure a smooth transition. Python 3 has been available on all our systems for some time, and it comes with nearly all the extra modules present in our Python 2 installations. This includes interfaces to all our in-house packages and libraries such as ecCodes, Magics or ecFlow. Metview's Python interface has been always only available in Python 3.

We would like to encourage everyone still using Python 2 to take action and start the migration as soon as possible. There are tools that can help you in this task, by doing most of the code changes for you. Please have a look at the Migration to Python 3 wiki page for some advise on how to do this.

Please do not hesitate to contact servicedesk@ecmwf.int for any questions or concerns.

Throughout the years, new software packages and multiple versions of them have been deployed on our different computing platforms. In preparation for our future migration to Bologna - New Data Centre, we will be streamlining the number of software packages and their versions available, so the majority of the user base is on recent software stacks prior to our move.

To accomplish this, we will extend our annual cleanup cycle, removing in a first stage the modules for unused versions as well as those ones that were never set as default. After the session, you will not be able to load the modules affected.

Some of those modules are still actively used, so you are encouraged to check if any of the packages in the list below is used anywhere in your processing workload. If so, we recommend moving to the current default version for each of those software packages and libraries.

Note that the actual deployments and installations for those software packages are not going to be deleted at this point. This will be done at a later stage. 

If you have any concerns, please get in touch with our servicedesk@ecmwf.int.

When?

The session is scheduled for Wednesday 18 September 2019 at 10:00 UTC

Where?

The platforms affected are:

• All clusters in our High Performance Computing Facility, mainly CCA and CCB
• ECGATE
• Linux Clusters
• Linux Desktops

Packages to be retired

These are the packages and versions that will no longer be available through the module system on the different platforms:

As part of the retirement of EMOSLIB, access to the current location of the BUFR tables is to be closed. When this happens, any existing program or tool using EMOSLIB for the BUFR decoding/encoding will stop working. This should raise awareness of the places in your workflow that need your attention.

When?

The session is scheduled for Wednesday 11 September 2019 at 10:00 UTC

Where?

The platforms affected are:

• All clusters in our High Performance Computing Facility, mainly CCA and CCB
• ECGATE
• Linux Clusters
• Linux Desktops

Actions to take

If you are currently using emoslib for the BUFR decoding, you will need to migrate those to ecCodes at some point. However, to continue working with the existing tools or code base as they are, there are two alternatives:

Recommended option

Rebuild your program with the new standalone BUFRDC package. This is the preferred option to make sure your program works well with the package that will be installed in the future platforms. Note that you 

module load bufrdc

If in a rush or causing operational problems...

You may temporarily keep the existing executables, libraries and tools as they are, but you need load the emos_bufr_tables module before running them. This will make the BUFR tables available for them. 

module load emos_bufr_tables

ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

Table of contents

Highlights

Software stack ready for next operational IFS cycle

With the upcoming Implementation of IFS cycle 46r1, this set of defaults is ready to handle the data produced, including all the new parameters introduced in this cycle.

Default GNU compiler updated

The GNU GCC compiler is used by default in our Linux Clusters. After this session, version 7.3 of this popular compiler will be used by default, instead of the old 6.3.

Python 3 Ready

A more complete Python 3 deployment has been made, with all major ECMWF packages supporting it: Magics, Metview, ecFlow and ecCodes (in beta). We encourage all Python users to start migrating their programs to Python 3, as the official support for Python 2 will cease at the end of 2019. Please have a look at the Migration to Python 3 documentation page for more details on how to do this migration.

Metview new features

The new Metview 5.5 features a new set of thermodynamic functions and lots of new functionality for the Geopoints format.  It also uses the new interpolation package MIR by default, replacing the old EMOSLIB interpolation library.

See the Magics and Metview latest news for a more detailed list of changes.

Improved ecFlowUI

This ecFlow release introduces the Timeline View in ecFlowUI. This is available as a tab in the Info Panel and provides a graphical view of node state history taken from the log file, allowing greater overview and analysis of node start and run times. It also supports a Duration view mode, displaying the submission and active times. This mode allows you to filter the nodes of your choice. The Timeline View provides the ideal tool to help you identify and optimise the tasks that are taking too much time.

Update of main third-party software package defaults across all platforms

Once a year we take the opportunity to update not only the software we develop at ECMWF, but also the core 3rd party packages and libraries in our software stack. Amongst the affected packages we can find popular tools and libraries like Python, R,  cdo or netCDF 4. See below for the complete list.

Discontinuation of old PGI compilers

Up to now, we have been providing our software libraries built with a legacy PGI 14 compiler for the FORTRAN parts of the code. This helped with the migration from older platforms to new ones where our default compiler of choice has been the GNU GCC. The new versions becoming default now are no longer available for the PGI environment on those platforms where it is present. If you still use it, please consider moving to the default environment. Note that older versions of our software stack will still be available for a bit longer in case you cannot migrate immediately.

How to use the new versions for testing BEFORE the update

The new versions of the software packages can be selected by using the jun19 tag in modules. For example:

module swap metview/jun19
module load Magics/jun19
module swap eccodes/jun19 
module swap python/jun19

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

module swap metview/old
module load Magics/old
module swap eccodes/old 
module swap python/old

Ecflow change: possible impact after upgrading

This release corrects some faulty triggers expression, that may require validation:

...
family f1
   task t1
       trigger f1 == aborted
   task t2
       trigger ./f1 == aborted

...
family f1
   task t1
        trigger ../f1 == aborted
   task t2
        trigger ../f1 == aborted

If your suite contains wrongly defined triggers like the one on the left, they will never be satisfied and therefore the tasks or families depending on them will not run.

It is highly recommended that whenever you migrate to version 4.12 or beyond from an earlier release, you run the checks below. This will highlight problems early, and save you a lot of debug time.

This can be done before we migrate.

•  If you use python to generate your suite definitions, then simple make sure you run(defs.check())

  module load ecflow/ new ... # build your python suite definitions def = ... print defs.check()  #  this will print any problems with triggers and inlimit

• Alternatively you can use:

  module swap ecflow/ 4.12.0 ecflow_client --get > tmp.def ecflow_client --load tmp.def check_only #  this will print any problems with triggers and inlimit

If you have migrated to the new release, to check all your suites run:

You have all the details in the ecflow release 4.12.0.

Change list

ECMWF Software

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

Third-party software

There are many modules of ECMWF and 3rd-party software deployed on all platforms at the centre that, according to our logs, haven't been used for long time. In a first stage in November, we removed the modules for those unused versions.  Now is time to close the access to the actual deployments and installations for those software packages and libraries. Although they are not going to be deleted at this point, they will become unavailable.

If you have any concerns, please get in touch with our servicedesk@ecmwf.int.

When?

The session is scheduled for Wednesday 13 February 2019 at 10:00 UTC

What and where?