Blog from October, 2020

ECMWF is updating the default versions of its software packages and libraries used across all user platforms at the centre as part of the regular update cycle.

When?

The changes will take place on Wednesday 25 November 2020 10:00 - 10:30 UTC

What platforms are affected?

All user platforms at the centre will see their default additional software stack updated:

  • All clusters in our High Performance Computing Facility, mainly CCA and CCB
  • ECGATE
  • Linux Clusters
  • Linux Desktops

The next defaults are tagged as "nov20" in modules. After the session the previous defaults will be tagged as "old"

Table of contents

Highlights

A number of bug fixes and performance improvements have been introduced on the different versions to become default. Here are some of the most relevant new features in respect to the previous default set of versions:

  • ecCodes, the heart of GRIB and BUFR handling, brings performance improvements, new parameters, contributions from users and several smaller features/bug fixes including support for the Mercator projectionPlease note that ecCodes' Python 2 interface is no longer being developed. Users are strongly advised to migrate to Python 3.
  • Magics, ECMWF's plotting library, adds some smaller fixes and additional improvements including :
    • combined with the latest ecCodes, Magics brings plotting support, including interactive Cursor Data, for GRIBs encoded with various projections such as Mercator and Lambert variants.
                 
    • a major code clean-up has reduced the number of dependencies. The Boost library and Python are not required to install Magics anymore.
    • fix issue with logging that was degrading observation plotting speed when used from Python
    • fix issue in python when passing single element array.
  • Metview, ECMWF's interactive and batch software for manipulation and plotting of data, brings several new features and functions, such as:
    • much improved speed of plotting in-memory arrays of data using the Input Visualiser thanks to allowing vectors (Macro language) or numpy arrays (Python) to be used as data
    • the last two modules that were originally implemented in Fortran (Spectra and Potential Temperature) have been converted to C++, meaning they will be available to more users
    • fixes for macOS
    • new Gallery examples
    • a new regridding module that exposes many new interpolation methods for expert users
    • user interface improvements such as a new preview panel that allows quick inspection of icons
    • the long-awaited ability to filter for parameters in the icon editors makes its way into this release
    • a whole raft of new Python/Macro functions for thermodynamic computations and plotting utilities
    • more - see the release notes!

      Metview's new Regrid module brings new possibilities in gridding and transformation of data

  • CodesUI, the standalone combination of Metview's GRIB and BUFR examiners, added syntax highlighting for textual GRIB dumps and fixed an issue where the BUFR filter could unexpectedly crash.

How to use the new versions for testing BEFORE the update

Users are strongly encouraged to test their software applications and data processing chain with the new versions of the various software packages before the date of the change to the default versions.

The new versions of the software packages can be selected by using the nov20 tag in modules. For example:

module swap metview/nov20
module load Magics/nov20
module load bufrdc/nov20
module swap eccodes/nov20 

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

module swap metview/old
module load Magics/old
module load bufrdc/old
module swap eccodes/old 

Please let ECMWF know if you need to revert to the old version of any of the packages after the update of the defaults has taken place. 

Version changes

This table summarises the changes in the version of the different packages:

Please see the Release notes for full details of the changes implemented in each package.

ecCodes is now the official ECMWF encoder and decoder for GRIB and BUFR formats. It superseded GRIB-API, for the GRIB format, and EMOSLIB, for the BUFR. If you still use any of those, we encourage you to migrate to ecCodes. EMOSLIB is, in fact, no longer updated. The standalone BUFRDC package is also installed to help in the migration.

PackageDefault version
BeforeAfter
BUFRDC4.6.34.6.4

Caveats

ecCodes 2.19 and Snow Depth Parameter (141)

ParameterNumber for Snow Depth (141) is now 254 instead of 11 when encoding in GRIB2 format.

The GRIB edition 2 for 141 encoding was wrong in previous versions of ecCodes: The parameterNumber should be 60, not 11 because 11 is the physical depth not the one converted to water volume.

In older versions, both paramIds 141 and 3066 shared the same GRIB2 representation and the consequence of that was 141 was obfuscated by 3066 (water converted depth is obfuscated but the true physical depth!)

From ecCodes 2.19 onwards, this parameter uses local GRIB2 encoding in ECMWF local table 1:

discipline = 0;
parameterCategory = 1;
parameterNumber = 254;
localTablesVersion = 1 ;