Blog

ecFlow 4.17.0 released

Version 4.17.0 of ecFlow, ECMWF's workflow management software has been released. It brings a number of improvements in the GUI including improved rendering of date and time attributes that are holding a node, and the ability to kill consumers of a limit directly from the limit's dialogue. See ecFlowUI Release Notes for full details. Among the improvements in the server are an update to the 'repeat date' generated variables, which can now be correctly referenced from trigger expressions for fine-tuned date-based triggering. Another fix in the server allows child tasks that are scheduled to run on a particular day to continue running even if their predecessor ran over the midnight boundary.

The October 2019 release of our software provides many fixes, improvements and new features!

ecCodes, the heart of GRIB and BUFR handling, brings its usual support for new GRIB parameters along with numerous fixes for extra stability. There are performance improvements for decoding/encoding GRIB data in a multi-threaded program as well as new options for the grib_ls and grib_dump tools.

Metview, ECMWF's interactive and batch software for manipulation and plotting of data, brings several new Macro/Python functions, including ml_to_hl(), which converts GRIB data from model levels to height above ground or sea level. Also in this release is an improved inline Help sidebar in the Code Editor.

CodesUI, the standalone combination of Metview's GRIB and BUFR examiners, added new tab called Descriptors to show all the expanded descriptors (metadata only) of the given message.

Magics, ECMWF's plotting library, adds some smaller fixes.

image2019-10-10_8-45-24.png      

Metview's Code Editor offers a list of available functions and can open a web browser tab to the right place in the documentation for each one

Metview, Magics and ecCodes are all now available on conda through the conda-forge channel. Another way to get the binary installations is via RPM, e.g. through the links here.

Please note that when building from source, the minimum CMake version requirement is 3.6.

Metview is built at ECMWF with all the versions of our libraries listed in the table below, and all are included in the October 2019 Metview Bundle. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

July 2019 software release

This release brings many exciting changes! Metview brings a number of improvements to its user interface, including improved inline help in the Code Editor and a new way to drop icons. In addition to many improvements to the Macro/Python functions, we also added the much-requested feature to allow the plotting of user-supplied shapefiles as part of the background. If you dig around the documentation, you'll also find some new pages such as the Icon Reference and the Data Types landing pages and 8 new Gallery examples, all designed to make Metview's capabilities more accessible.

ecCodes introduces a new Python 3 interface, available on PyPi (pip3 install eccodes-python). This is the way to install the Python 3 bindings; legacy Python 2 support is still available through the bindings that are built as part of the ecCodes binary package.
Also ecCodes now incorporates the contribution from Météo-France to support limited area model fields for AROME and ALADIN.

Metview can now plot a user-supplied shapefile in a geographic plot

Please note that we have upgraded the minimum CMake version requirement to 3.6.

Metview is built at ECMWF with all the versions of our libraries listed in the table below, and all are included in the July 2019 Metview Bundle. All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

With the February 2019 synchronised release, for the first time we make the popular ECMWF software packages ecCodes and Magics available under the Windows operating system. Our partner Daniel Tipping from Old Reliable Tech made the necessary changes in both packages in order to compile under Windows and packaged them with the widely used pip and conda package managers of the Python community. This development allows users to decode and visualise BUFR and GRIB data on Windows using these tools.

Windows is not a platform that is used for technical work at ECMWF. Therefore we will not be able to support this platform as well as we do for others, such as Linux. If you find problems, we invite the Windows user community to use our new set-up on GitHub to fork/contribute and issue pull request changes to ecCodes and Magics. Old Reliable Tech has set up automatic tests and builds for Windows on GitHub, which should ensure continued releases for this new platform.

Please let us have your feedback on this work!

Snapshots showing Magics visualising GRIB fields decoded by ecCodes (right) and Magics showing NetCDF data (left). Both within Jupyter notebooks running under Windows.

This month's software release sees many major changes. This release should bring many improvements for Python users, with core packages, such as ecCodes and Magics now being accessible through pip and conda. This also includes now the Windows operating system. With version 4.0 of Magics the Python interface is now not part of the the core software package anymore. Users should use pip or conda to install the Python extensions.  This has the benefit of allowing support for using multiple Python versions at the same time. Magics also contains improvements for running with Jupyter notebooks. For examples see our Magics Jupyter notebook examples.

This is the first release in which Metview uses the new interpolation package MIR. We recommend users to test this version carefully before introducing this in production use. The previous interpolation through Emoslib is still possible. We welcome any feedback to this change. Metview 5.5 also features a new set of thermodynamic functions and lots of new functionality for the Geopoints format.

 

New features of Metview: compute and plot a parcel path in a thermodynamic diagram (left) & 
functions based on the horizontal derivatives of regular lat-lon fieldsets have been added (right)

Metview is built at ECMWF with all the versions of our libraries listed in the table below, and all are included in the February 2019 Metview Bundle. All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

ecCodes2.12.0ecCodes version 2.12.0 released
Emoslib4.5.9

no update this months

ODB API0.18.1no update this months
Magics4.0.0Latest News
Metview5.5.0Version 5.5 Updates
ecFlow 4.12.0 released

The newly released ecFlow 4.12.0 introduces a number of new features (ecflow release 4.12.0).

In the future, support for Python 2 will be dropped and only Python 3 will be supported. To aid this effort, ecFlow now supports both Python 2 and 3 modules in the same release. This functionality can be enabled if ecFlow is built using CMake 3.12.0 or above. At ECMWF, ecFlow is now built with support for both Python 2.7 and Python 3.6.

This release also introduces the Timeline View in ecFlowUI. This is available as a tab in the Info Panel and provides a graphical view of node state history taken from the log file, allowing greater overview and analysis of node start and run times. It also supports a Duration view mode, displaying the submission and active times. This mode allows you to filter the nodes of your choice. The Timeline View provides the ideal tool to help you identify and optimise the tasks that are taking to much time.

  

This release  allows us to list, the set of nodes that depend on a Event.(i.e. trigger references to an event).   First select the event in ecflowUI, then with RMB select  'Edit ...'


The script location algorithm (i.e. how the server finds your .ecf/.sms files) is now more flexible. Locating ‘.ecf’ file

 

This release corrects some faulty triggers expression, that may require validation.

It is highly recommended that whenever you migrate from one release to another, you run the checks(see below) on the new release.

This will highlight problems early, and save you a lot of debug time.

This can be done before we migrate.

  •  If you use python to generate your suite definitions, then simple make sure you run(defs.check())

    module load ecflow/new
    ...
    # build your python suite definitions
    def = ...
    print defs.check()  # this will print any problems with triggers and inlimit

  • Alternatively you can use:

    module swap ecflow/4.12.0
    ecflow_client --get > tmp.def
    ecflow_client --load tmp.def check_only # this will print any problems with triggers and inlimit

  • If you have migrated to the new release, to check all your suites run:

    module load ecflow/4.12.0
    ecflow_client --check=_all_   # this will print any problems with triggers and inlimits



We are happy to announce the December versions of the standard ECMWF software packages. This month's release featured mostly incremental updates to our packages. As announced, this is the very last release of GRIB_API. We encourage users to migrate to ecCodes.

The default set of tests in ecCodes do not require downloading of data anymore. We are aware that many maintainers/packagers of ecCodes did not run tests to avoid these downloads. Please enable the tests again, they are vital to ensure ecCodes has been installed properly. Anyone developing/changing code in ecCodes should run the extended set of tests which requires the data download, but exercises the ecCodes functionality more thoroughly. This release contains the latest WMO BUFR tables (November update) with ecCodes and Emoslib/BUFRDC.

Metview offers a large selection of layers and automatic styles based on the ecCharts web map service via its new ecCharts icon.

  

Metview now offers the ecCharts layers and styles (left), which are searchable and previewed in the icon editor (right).

Python 3 interface for ecCodes

With ecCodes 2.10.0, we prepared a beta version of the long awaited Python 3 interface to the ecCodes API. While the migration to Python 3 of many other packages has been done, ecCodes had proved more challenging. The main challenge has been that the underlying file I/O has radically changed with Python 3, therefore we needed to rewrite some of the underlying code. This required careful testing to ensure no regression in functionality (including the Python 2 installations). We try to minimise the impact to users and their programs, but some minor changes required by Python 3 itself are needed. When opening any BUFR or GRIB files please ensure this happens in binary mode, like this:

= open("myfields.grib", "rb")

This change in your programs will also work in Python 2. We are also aware of issues if GRIB/BUFR files are reopened in the same programs and will investigate this issue. We would like to thank everyone who expressed an interest and support. Thank you for your patience. Anyone interested in a higher level Python 3 interface to read GRIB data into xarray, should have a look at cfgrib. Please test this Python3 interface carefully and provide us with feedback.


All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

We are happy to announce we have released the October versions of the standard ECMWF software packages. This month's release featured mostly incremental updates to our packages. The only major change is to Magics, where internally the Boost code was replaced, reducing the dependencies of the library and speeding up the compilation of the library. While much testing has been done and much care been taken, there might be still some impact on the contouring. Please let us know if you might see any regressions. ecCodes saw many smaller improvements, especially for the performance of BUFR handling. Metview offers new automatic styles based on the ecCharts web map service.

  

Metview now offers the ecCharts styles (left), which are previewed in the icon editor (right).

All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

Metview 5.1.1 released

A Metview hotfix version has been released, including fixes for geopoints-to-grib conversion, the Metview Python interface and some build issues.

See Version 5.1 Updates for more details.


We can announce that this weeks we released the June versions of the standard ECMWF software packages. This month's release sees with Metview 5.1 and Magics 3.1 the use of ecCodes for BUFR decoding rather than BUFRDC. This eliminates for Magics the dependency on Emoslib. Magics is also now independent of Perl and only requires Python for generating codes. With the change to ecCodes for BUFR decoding, the GRIB and BUFR examiners in Metview were rewritten. There are also performance improvements and fixes in the grib_to_netcdf tool in ecCodes.

Metview added the set of predefined palettes available in Magics (see the example below). For more information on these, please see Predefined palettes in Magics for the list of available palettes.

 thermo-plots-x3.png

The new BUFR examiner in Metview, based on ecCodes (left) and new thermodynamic diagram types in Metview (right).

New automatic styling available in Magics and Metview.

Metview is built at ECMWF with all the versions of our libraries listed in the table below, and all are included in the June 2018 Metview Bundle. All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

We have made a special hotfix release for ecCodes to add new GRIB lightning parameters used in the upcoming IFS cycle 45r1.
Note: This update is only necessary if you wish to use this cycle's new lightning parameters.

This version is available on all ECMWF computer systems as "new". If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

First Release of CodesUI

End of February saw the first external release of CodesUI, an interactive application built on ecCodes to inspect GRIB and BUFR messages. With CodesUI it is possible to inspect the structure of GRIB and BUFR files and examine the metadata and data of individual messages. For BUFR advanced filtering and data extraction (see the examples here) is available together with the plotting of observation locations on an interactive map.

The structure of a BUFR messageInteractive location map for BUFR observations

With all these capabilities CodesUI can be regarded as a replacement for BUFR ToolBox, a legacy Java-based application built on the old BUFRDC library, used by many at ECMWF to work with BUFR data.

Starting up CodesUI from a terminal at ECMWF (installed on the Linux Desktops and ecgate) is really simple, just type this to inspect any GRIB or BUFR files:

module load codes_ui
codes_ui your_bufr_or_grib_files   # wildcards accepted, e.g. *.bufr

CodesUI was developed to be both a component of Metview and a standalone software package with the minimum possible dependencies, in fact to build CodesUI only ecCodes and Qt5 are needed.

CodesUI will be the basis for the new GRIB and BUFR examiners in the forthcoming Metview release 5.1.0, which will use ecCodes for all its BUFR handling including its Observation Filter and Bufr Picker modules. We plan to add an even more advanced BUFR filter module to Metview, based on that in CodesUI, later this year.

A bit late (wink), we can announce that two weeks ago we released the February versions of the standard ECMWF software packages. Last month's release featured the first full version of Metview 5 and minor updates to the other libraries. Please see the December 2017 Software release blog item below for more information about new features of  Metview 5.0 and Magics 3.0.

image2017-11-9_10-59-5.png

Metview is built at ECMWF with all the versions of our libraries listed in the table below, and all are included in the February 2018 Metview Bundle. All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions. If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.

This week saw the December 2017 release of the standard ECMWF software packages. This month's release showcases the release of the first beta version of Metview 5. This brings some substantial new features such as the ability to edit contouring styles directly from the interactive plot window, new methods to generate colour palettes and the integration of the FLEXPART particle dispersion model. Also released is Magics 3.0.0, which provides the backbone of Metview's plotting features, and now offers a Python 3 interface. Since these are major updates, we recommend that users test these versions carefully. There are also updates in ecCodes and ODB_API. Note that emoslib has simply been relinked with the latest ecCodes and contains no code changes itself.

Metview 5's interactive plot editing

Transparency features in Metview 5

FLEXPART output visualised with Metview 5

Metview is built at ECMWF with all the versions of our libraries listed in the table below, and all are included in the December 2017 Metview Bundle.
All these packages use the same build system based on CMake - simplifying and harmonising the installation experience. The versions are available on all ECMWF computer systems as "new" versions, apart from Metview, whose "new" version will be updated after New Year; for now, it is available via the module "metview/5b". If you encounter any issues please feel free to send feedback to Software.Support@ecmwf.int.


Reminder: Starting with the November 2016 release all packages now use ecCodes as their default library to decode/encode GRIB data, instead of GRIB_API. While we are confident that ecCodes is to a high degree backwards compatible with GRIB_API, we strongly recommend good testing before using the software in your operational environments. In case you prefer to continue using GRIB_API for now, we provide a minor update release of this package, but recommend that you move to ecCodes eventually, since new features will only be implemented in ecCodes.

Software e-learning modules

The first batch of ECMWF's new e-learning modules has been released! On the software side, these include modules that introduce ecCodes, MARS and Metview.

For more information, go to: