ECMWF updated the default versions of its software packages and libraries used on Atos HPCF and ECS as part of the regular update cycle.

When?

The changes took place on Wednesday 15 May 2024 09:00 UTC

Do I need to do anything?

We strongly recommend you test that your programs, scripts and suites work with the new versions of any software or library affected before the change happens.

Table of contents

Highlights

Python 3.11 becomes default

For our Python users, we now offer a more recent version of Python 3.11 with the usual extra modules found on the previous default 3.10. Note that new versions of other packages with python bindings may only come with support for the new Python 3.11.

Enhanced experience for the new ECMWF Jupyterhub service

We have recently opened our new ECMWF Jupyterhub service at https://jupyterhub.ecmwf.int. to all users of our HPCF and ECS.  While the old software stack is also supported, you will benefit from an updated Jupyter environment and tools. You can find more information about this new way of working on the HPCF and ECS in the ECMWF JupyterHub documentation pages.

ECMWF software improvements

There have been a vast number of improvements and changes in our ECMWF software and libraries since our last default change in November 2023. See the blog posts for all the software releases since then to learn about what is new in ecCodes, Magics or Metview.

ecCodes changes encoding for certain GRIB2 parameters and other changes

Since version 2.28.0, ecCodes has changed the way it encodes certain GRIB2 parameters, in preparation for our Migration to GRIB edition 2. The complete list can be found in Migration to GRIB2 - new in ecCodes . If you encode those particular GRIB2 fields with a more recent ecCodes version, you will not be able to decode them properly with an older ecCodes. We strongly recommend to make sure all your production chain uses an up-to-date ecCodes version for best results.

Additionally, the key "levtype" is now read-only (for GRIB edition 2). You should use "typeOfLevel" to set fixed surface types. 

For consistency, the key "validityTime" now has 4 characters as a string like dataTime. This affects the output of "grib_ls -n time"

A lot of other bugfixes and improvements

All the new versions, both ECMWF and 3rd Party, come with a number of bug fixes and improvements. You may see the change logs and news for any individual package or library in the list below if you want to have all the details.

How to use the new versions for testing BEFORE the update

Users are strongly encouraged to test their software applications and data processing chain with the new versions of the various software packages before the date of the change to the default versions.

The new versions of the software packages can be selected by using the may24 tag in modules. For example:

module load ecmwf-toolbox/may24
module load python3/may24

How to revert to the old versions in case of problems AFTER the update

Users that experience problems with the new default software packages after the update will be able to revert to the old default versions. These can be selected with the commands:

module load ecmwf-toolbox/old
module load python3/old

Please let ECMWF know if you need to revert to the old version of any of the packages after the update of the defaults has taken place. 

Change list

ECMWF Software

The new version of the ECMWF Toolbox (ecmwf-toolbox module) is 2024.04.0.0, replacing 2023.10.1.0. This table summarises the changes in the version of the different packages within the bundle:

PackageDefault versionRelease notesDownload page
BeforeAfter
CodesUI1.7.41.8.0CodesUI - Change HistoryCodesUI - Releases
ecCodes2.32.12.35.0

Latest news

Migration to GRIB2 - new in ecCodes 

ecCodes Releases
Magics4.14.14.15.4Latest NewsMagics Releases
Metview5.20.05.22.0

Release notes

Metview Releases

ODC1.4.6

1.5.0

ReadmeODC Releases

Besides ecmwf-toolbox, the following ECMWF packages with their corresponding module also see their default changed:

ecFlow5.11.45.12.4

Latest news

ecFlow Releases
climetlab0.12.8

0.21.3

https://github.com/ecmwf/climetlabhttps://github.com/ecmwf/climetlab/releases

Please see the Release notes for full details of the changes implemented in each package.

Third-party software

Some of the versions in this list may already be the default for a specific package, so no changes will be made in those cases.

ModuleBeforeAfterRelease notes
adios2.9.22.9.2https://adios2.readthedocs.io/en/latest/
aec1.0.6

1.1.2

https://gitlab.dkrz.de/k202009/libaec/-/blob/master/CHANGELOG.md
amd-fftw3.0.14.1https://github.com/amd/amd-fftw/blob/amd-fftw/ChangeLog
ansible7.1.0

9.2.0

https://github.com/ansible-community/ansible-build-data/blob/main/7/CHANGELOG-v7.rst
antlr2.7.72.7.7https://www.antlr2.org/doc/
aocc4.0.04.0.0https://www.amd.com/en/developer/aocc.html
apptainer1.1.81.3.0https://github.com/apptainer/apptainer/blob/main/CHANGELOG.md
bison3.8.23.8.2https://fossies.org/linux/bison/ChangeLog
blis3.13.2https://github.com/amd/blis/blob/master/CHANGELOG
boost1.81.01.84.0https://www.boost.org/users/history/version_1_84_0.html
cdo2.1.12.4.0https://code.mpimet.mpg.de/projects/cdo/news
cgal5.5.25.6https://www.cgal.org/releases.html
cmake3.25.23.28.3https://cmake.org/cmake/help/latest/release/
cmor3.7.13.7.3https://cmor.llnl.gov/news/
coda2.24.12.25.1https://github.com/stcorp/coda/blob/master/CHANGES
conda22.11.1-223.11.0-0https://docs.conda.io/projects/conda/en/latest/release-notes.html
cppcheck2.92.13.0https://github.com/danmar/cppcheck/releases
darshan3.4.23.4.4https://www.mcs.anl.gov/research/projects/darshan/
doxygen1.9.61.10.0https://doxygen.nl/manual/changelog.html
eigen3.4.03.4.0https://eigen.tuxfamily.org/index.php?title=3.4
esmf8.4.08.6.0https://earthsystemmodeling.org/static/releases.html
extrae3.8.04.0.6https://github.com/bsc-performance-tools/extrae/blob/master/ChangeLog
fcm2021.05.02021.05.0https://github.com/metomi/fcm/blob/master/CHANGES.md
fdupes2.2.12.2.1https://github.com/adrianlopezroche/fdupes/releases
ferret7.6.37.6.3https://github.com/NOAA-PMEL/PyFerret/releases
ffmpeg5.16.1.1https://github.com/FFmpeg/FFmpeg/blob/master/Changelog
fftw3.3.103.3.10https://www.fftw.org/release-notes.html
fgsl1.5.01.5.0https://github.com/reinh-bader/fgsl/blob/master/NEWS
flex2.6.42.6.4

https://github.com/westes/flex/releases

gdal3.6.23.8.4https://github.com/OSGeo/gdal/blob/v3.6.2/NEWS.md
gdl1.0.21.0.4https://github.com/gnudatalanguage/gdl/releases
geocat2022.11.02024.01.0https://github.com/NCAR/geocat-comp
geos3.11.1

3.12.1

https://github.com/libgeos/geos/releases
git2.39.12.43.1https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.39.1.txt
gitflow1.12.31.12.3https://github.com/petervanderdoes/gitflow-avh/releases/tag/1.12.3
github-cli2.44.12.44.1https://github.com/cli/cli/releases
gmt6.4.06.50https://docs.generic-mapping-tools.org/dev/changes.html
gnuparallel20230122

20240122

https://www.gnu.org/software/parallel/
go1.19.51.12.7https://go.dev/doc/devel/release
gperf3.13.1

https://github.com/rurban/gperf/blob/master/ChangeLog

grace5.1.255.1.25https://plasma-gate.weizmann.ac.il/Grace/
grads2.1.a2.oga.12.1.a2.oga.1http://opengrads.org
grass8.2.18.3.1https://grass.osgeo.org/news/2023_01_21_grass_gis_8_2_1_released/
gsl2.72.7http://git.savannah.gnu.org/cgit/gsl.git/tree/NEWS
hdf4.2.144.2.14https://www.hdfgroup.org/2018/07/release-of-hdf-4-2-14-newsletter-162/
hdf5-parallel1.12.21.14.3https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2-RELEASE.txt
hdf51.12.21.14.3https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2-RELEASE.txt
hdfeos3.03.0https://wiki.earthdata.nasa.gov/display/DAS/Toolkit+Downloads
hdfview3.1.43.3.1https://www.hdfgroup.org/downloads/hdfview/
hpcx-openmpi2.9.02.9.0https://developer.nvidia.com/networking/hpc-x
iris3.4.03.7.0https://scitools-iris.readthedocs.io/en/v3.4.1/whatsnew/index.html
jasper4.0.04.2.0https://github.com/jasper-software/jasper/releases
java11.0.611.0.6https://www.oracle.com/java/technologies/javase/11all-relnotes.html
julia1.8.51.10.0
lcov1.162.0https://github.com/linux-test-project/lcov/releases
lftp4.9.24.9.2https://lftp.yar.ru/news.html
libflame3.13.1https://github.com/flame/libflame
libspatialindex1.9.31.9.3https://github.com/libspatialindex/libspatialindex/releases
libtensorflow2.7.02.15.0https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md
maven3.8.73.9.6https://maven.apache.org/release-notes-all.html
mongodb5.0.87.0.5https://www.mongodb.com/docs/manual/release-notes/
nccmp1.9.1.01.9.1.0https://gitlab.com/remikz/nccmp
ncl6.6.26.6.2https://github.com/NCAR/ncl
nco4.9.75.1.9https://nco.sourceforge.net/highlights_old.shtml
ncview2.1.72.1.8http://meteora.ucsd.edu/~pierce/ncview_home_page.html
netcdf4-parallel4.9.14.9.2https://docs.unidata.ucar.edu/netcdf-c/current/RELEASE_NOTES.html
netcdf44.9.14.9.2https://docs.unidata.ucar.edu/netcdf-c/current/RELEASE_NOTES.html
ninja1.11.11.11.1https://github.com/ninja-build/ninja/releases
node18.13.020.11.0https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#18.13.0
numdiff5.9.05.9.0https://www.nongnu.org/numdiff/
nvidia22.1124.1https://developer.nvidia.com/hpc-sdk
octave7.3.08.4.0https://octave.org/NEWS-7.html
onnxruntime1.10.01.16.3https://github.com/microsoft/onnxruntime
openblas0.3.210.3.26https://github.com/xianyi/OpenBLAS/releases
openjpeg2.5.02.5.0http://www.openjpeg.org/2022/05/13/OpenJPEG-2.5.0-released
panoply5.2.35.3.2https://www.giss.nasa.gov/tools/panoply/changelog.html
parmetis4.0.34.0.3http://glaros.dtc.umn.edu/gkhome/metis/parmetis/changes
patchelf0.17.20.17.2https://github.com/NixOS/patchelf/releases
pcraster4.4.0-014.4.1-01https://pcraster.geo.uu.nl
petsc3.20.13.20.3https://gitlab.com/petsc/petsc
pnetcdf1.12.31.12.3https://github.com/Parallel-NetCDF/PnetCDF/blob/master/RELEASE_NOTES
postgres14.514.2https://www.postgresql.org
proj9.1.19.3.1https://proj.org/news.html
psutils1.171.17https://github.com/rrthomas/psutils
pycharm2022.3.22023.3.3https://www.jetbrains.com/pycharm/whatsnew/
pypy3.9-v7.3.11-01

3.10-v7.3.15-01

https://doc.pypy.org/en/latest/index-of-release-notes.html
python33.10.10-013.11.8-01https://docs.python.org/3.10/whatsnew/changelog.html
qgis3.28.33.34.3https://www.qgis.org/en/site/forusers/visualchangelogs.html
qhull8.1-alpha18.1-alpha1http://www.qhull.org/news/qhull-news.html
qt5.12.06.6.1https://wiki.qt.io/Main
R4.2.24.3.3https://cran.r-project.org/doc/manuals/r-release/NEWS.html
rclone1.61.11.65.2https://rclone.org/changelog/
rstudio2022.12.0+3532023.12.1+402https://docs.posit.co/ide/news/
s3cmd2.3.02.1.0https://github.com/s3tools/s3cmd/releases
snakemake7.32.38.5.5https://snakemake.readthedocs.io/en/stable/project_info/history.html#changelog
sqlite3.40.13.46.0https://www.sqlite.org/changes.html
sqlitebrowser3.12.23.12.2https://github.com/sqlitebrowser/sqlitebrowser/releases
stat4.2.14.2.1https://github.com/LLNL/STAT
texlive20222023https://www.tug.org/texlive/
udunits2.2.282.2.28https://github.com/Unidata/UDUNITS-2
wgrib1.8.31.8.5https://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html
wgrib23.1.23.1.3https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/
wxparaver4.9.24.11.3https://github.com/bsc-performance-tools/wxparaver
xconv1.941.94https://ncas-cms.github.io/xconv-doc/html/index.html

Potential issues

ecFlow and python3 incompatibilities

While the ecFlow client and python3 default versions are updated on the HPCF, if you may be running an older version of the ecFlow server. If in your ecFlow tasks you use the ecFlow Python bindings, they may not work with the new default version of Python.

We will update the version of all servers shortly, but in the meantime you may need to either revert to using python3/old in your tasks, or force them to load the new default ecFlow module.

  • No labels