Versions Compared

Old Version 13

changes.mady.by.user Michela Giusti

Saved on

compared with

New Version 14

changes.mady.by.user Michela Giusti

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Page info
infoTypeModified date
prefixLast modified on
typeFlat
Info
iconfalse
titleTable of Contents
Table of Contents
maxLevel2
excludeGRIB to NetCDF4 conversion in the CADS|Get example grib data from the CDS|Configuration options|Open the GRIB as a dictionary of xarray datasets|Modify the xarray object to meet the defined standard|Write the datasets to NetCDF

Introduction

The new CDS and ADS systems (currently the beta system) have a new GRIB to netCDF converter. This new converter is based on xarray using the cfgrib engine. As each dataset has it's own particular features, the configuration used differs from dataset to dataset to ensure that the returned results are correct and appropriately strcutured.

The general workflow of the new conversion is as follows (for a more detailed description, expand the Jupyter Notebook below):

  1. Open the GRIB file as an xarray.Dataset using the cfgrib engine
    1. Where necessary this is a list of xarray.Dataset such that the data is organised into complete and compatible hypercubes
    2. As each dataset has it's own specifics, the options used when opening the GRIB files differ from dataset to dataset. These options can be made available upon request, but an understanding of the Jupyter Notebook below is expected should users wish to make use of them.
  2. Rename dimension/coordinate variables to match those described in New Coordinate variable names section
    1. Additionally, dimensions are expanded to ensure that certain variables are also dimensions of the dataset, e.g. the valid_time dimension when there is a single time step.
  3. Store as netCDF4 file[s] (with internal compression) and return to the user
Note

As GRIB is the native format of most of the datasets produced by ECMWF, the format used in the netCDF files served by the CDS/ADS should not be considered a long-term stable format. Therefore, it is highly discouraged that these netCDF files are used in operational/downstream services as the standard is subject to change.

There are other openly available tools and software available for working with GRIB files, including conversion to netCDF. Please see the following resources for more guidance on using the GRIB files produce by ECMWF directly:

Why we are changing

There are a number of important reasons for the change in the netCDF produced by the CDS-Beta.

  • NetCDF4 is a more modern, more capable, and more future-proof version of netCDF. 
  • The legacy CDS and ADS used a mixture of 'grib_to_netcdf' (netCDF3) and cfgrib (netCDF4) to convert GRIB files to netCDF, rather than a single common form of netCDF.
  • Additionally if files were post-processed in the CDS toolbox (e.g. daily statistics), they were also converted to the common data model (CDS-CDM), which introduced additional differences in the netCDF structure and content. 
  • As such, the legacy CDS and ADS system supported 3 different netCDF conversions and standards, all of which had their own issues.

The new systems will use cfgrib for the conversion, with some modifications as documented below. The same converter is used for direct downloads and post-processed data (e.g. the daily statistics from ERA5 and ERA5-Land), hence users will have consistency in the various netCDF files received from the modernised (currently beta) portals.

It is important to understand that given the intrinsic differences between GRIB and netCDF there is not a one size fits all approach when converting GRIB to netCDF. We aim to provide users with some sensible default options, but given the code used to perform the conversion is open source and documented it is possible for users to fine tune the conversion to their requirements if needed.

Note

Please also note that these changes may mean that some software packages (e,g, GrADS, CDO) may not be able to open these new netCDF files without modification (e.g. changes to dimension order, dimension names etc.)

Anchor
jupiternotebook
jupiternotebook
Jupter Notebook demonstration

Expand
titleJupyter notebook demonstrating the new GRIB to netCDF conversion (advance Python)

GRIB to NetCDF4 conversion in the CADS

This notebook demonstrates how GRIB files are converted to NetCDF in the CADS such that users have full traceability of the processing chain, and can modify the process should they have different requirements for their netCDF format.

Code Block
languagepy
In [1]:
# Import libraries used
from typing import Any
import os
import xarray as xr
import cfgrib
import cdsapi

Get example grib data from the CDS

To make it interesting we are going to request atmospheric, soil level and ocean wave data. This will give us multiple challenges in terms of handling different level types and spatial grids.

Code Block
languagepy
In [2]:
dataset = "reanalysis-era5-single-levels"
request = {
    'product_type': ['reanalysis'],
    'variable': ['10m_u_component_of_wind', '10m_v_component_of_wind', '2m_dewpoint_temperature', '2m_temperature', 'mean_sea_level_pressure', 'mean_wave_direction', 'mean_wave_period', 'sea_surface_temperature', 'significant_height_of_combined_wind_waves_and_swell', 'surface_pressure', 'total_precipitation'],
    'year': ['1985'],
    'month': ['02'],
    'day': ['03'],
    'time': ['00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'],
    'data_format': 'grib',
    'download_format': 'unarchived'
}

grib_file = "data.grib"
if not os.path.exists(grib_file): # Don't download the file if it already exists
    client = cdsapi.Client()
    grib_file = client.retrieve(dataset, request).download("data.grib")

# Get the base name of the file to use in the output file names
fname, _ = os.path.splitext(os.path.basename(grib_file))


Configuration options

Now we set some options that we will use to open the grib file in xarray and perform some minor post-processing to produce our NetCDF file.

Code Block
languagepy
In [3]:
# Configuration options for opening the GRIB files as an xarray object,
#  these depend on the dataset your are working with, this example is for ERA5 single-levels

# The open_datasets_kwargs is a list of dictionaries, which is used to open the grib file into a list of
#  consistent hypercube which are compatible netCDF. There are a number of common elements, so we define a
#  common set and use them in each open_dataset_kwargs dictionary.
common_kwargs = {
    'time_dims': ['valid_time'],
    'ignore_keys': ['edition'],
    'extra_coords': {'expver': 'valid_time'},
    'coords_as_attributes': [
        'surface',
        'depthBelowLandLayer',
        'entireAtmosphere',
        'heightAboveGround',
        'meanSea'
    ]
}
open_datasets_kwargs: dict[str, Any] | list[dict[str, Any]] = [
    # To open the atmospheric variables
    {**common_kwargs, 'filter_by_keys': {'stream': ['oper']}, 'tag': 'stream-oper'},
    # To open the ocean-wave variables
    {**common_kwargs,'filter_by_keys': {'stream': ['wave']}, 'tag': 'stream-wave'}
]
# Keywords that are used to modify the xarray object after opening
post_open_datasets_kwargs: dict[str, Any] = {
    'rename': {
        'time': 'forecast_reference_time',
        'step': 'forecast_period',
        'isobaricInhPa': 'pressure_level',
        'hybrid': 'model_level'
    },
    'expand_dims': ['valid_time', 'pressure_level', 'model_level']
}

# Configuration options for writing the xarray object to a netcdf file,
#  These are the options used for all datasets in the CADS
# Keywords that are used when writing to netcdf
to_netcdf_kwargs: dict[str, Any] = {
    "engine": "h5netcdf",
}
# Compression options to use when writing to netcdf, note that they are dependent on the engine
compression_options = {
    "zlib": True,
    "complevel": 1,
    "shuffle": True,
}

print(
    f"Converting {grib_file} to netCDF files with:\n"
    f"  open_datasets_kwargs: {open_datasets_kwargs}\n"
    f"  post_open_datasets_kwargs: {post_open_datasets_kwargs}\n"
    f"  to_netcdf_kwargs: {to_netcdf_kwargs}\n"
    f"  compression_options: {compression_options}\n"
)
Out [3]:
Converting data.grib to netCDF files with:
  open_datasets_kwargs: [{'time_dims': ['valid_time'], 'ignore_keys': ['edition'], 'extra_coords': {'expver': 'valid_time'}, 'coords_as_attributes': ['surface', 'depthBelowLandLayer', 'entireAtmosphere', 'heightAboveGround', 'meanSea'], 'filter_by_keys': {'stream': ['oper']}, 'tag': 'stream-oper'}, {'time_dims': ['valid_time'], 'ignore_keys': ['edition'], 'extra_coords': {'expver': 'valid_time'}, 'coords_as_attributes': ['surface', 'depthBelowLandLayer', 'entireAtmosphere', 'heightAboveGround', 'meanSea'], 'filter_by_keys': {'stream': ['wave']}, 'tag': 'stream-wave'}]
  post_open_datasets_kwargs: {'rename': {'time': 'forecast_reference_time', 'step': 'forecast_period', 'isobaricInhPa': 'pressure_level', 'hybrid': 'model_level'}, 'expand_dims': ['valid_time', 'pressure_level', 'model_level']}
  to_netcdf_kwargs: {'engine': 'h5netcdf'}
  compression_options: {'zlib': True, 'complevel': 1, 'shuffle': True}

Open the GRIB as a dictionary of xarray datasets

This is handled differently depending on whether the kwargs are passed as a list of dictionaries, or a single dictionary.

If a list we iterate over each dictionary and return a dataset which matches it. These should include the filter_by_keys argument which will only select the fields which match the filter. This is used to split the grib file into complete hypercubes (which are compatible with NetCDF). We can then use the tag argument to label each hypercube dataset, and subsequent NetCDF file, with an appropriate name.

If a single dictionary, we first try to open directly with xarray.open_dataset as a single hypercube. If there is an inconsistency in the hypercube, then this will fail and we retry to open with cfgrib.open_datasets which will automatically split the grib into complete hypercubes in the form of a list of datasets. NOTE: One draw back here is that cfgrib does not provide the information as to how the grib file was split, so we cannot label the hypercubes appropriately, therefore we just label with an number.

Code Block
languagepy
In [4]:
# if open_datasets_kwargs is a list, then we open the grib file as a list of datasets
if isinstance(open_datasets_kwargs, list):
    datasets: dict[str, xr.Dataset] = {}
    for i, open_ds_kwargs in enumerate(open_datasets_kwargs):
        # Default engine is cfgrib
        open_ds_kwargs.setdefault("engine", "cfgrib")
        # The tag in the open_datasets_kwargs is used to name the dataset,
        # and subsequently the NetCDF file, if no tag is provided the index number is used
        ds_tag = open_ds_kwargs.pop("tag", i)
        try:
            ds = xr.open_dataset(grib_file, **open_ds_kwargs)
        except Exception:
            ds = None
        if ds:
            datasets[f"{fname}_{ds_tag}"] = ds
else:
    # We make sure that cfgrib raises an error if it encounters any issues, 
    # e.g. with inconsistent field dimensions
    open_datasets_kwargs.setdefault("errors", "raise")

    # First try and open with xarray as a single dataset,
    try:
        datasets = {
            f"{fname}": xr.open_dataset(grib_file, **open_datasets_kwargs)
        }
    except Exception:
        print(
            f"Failed to open with xr.open_dataset({grib_file}, **{open_datasets_kwargs}), "
            "opening with cfgrib.open_datasets instead."
        )
        # cfgrib.open_datasets returns a list of datasets, and it is not yet possible to sensibly tag them,
        # so we use the index number as the tag
        datasets = {
            f"{fname}_{i}": ds
            for i, ds in enumerate(
                cfgrib.open_datasets(grib_file, **open_datasets_kwargs)
            )
        }

Modify the xarray object to meet the defined standard

The raw output from cfgrib uses the native MARS-GRIB metadata keys for the dimensions and coordinates. We modify this such that the dimension and coordinate names are a little more user friendly, and easy to distinguish given overlaps with CF convention definitions.

To make these steps easier to work with we have defined stand-alone functions, then apply them as we iterate over the dictionary of datasets.

Code Block
languagepy
In [5]:
# Define a function to safely rename variables in an xarray dataset,
#  i.e. ensures that the new names are not already in the dataset
def safely_rename_variable(dataset: xr.Dataset, rename: dict[str, str]) -> xr.Dataset:
    """
    Rename variables in an xarray dataset,
    ensuring that the new names are not already in the dataset.
    """
    # Create a rename order based on variabels that exist in datasets, and if there is
    # a conflict, the variable that is being renamed will be renamed first.
    rename_order: list[str] = []
    conflicts: list[str] = []
    for old_name, new_name in rename.items():
        if old_name not in dataset:
            continue

        if new_name in dataset:
            rename_order.append(old_name)
            conflicts.append(old_name)
        else:
            rename_order = [old_name] + rename_order

    # Ensure that the conflicts are handled correctly
    # Is this necessary? We can let xarray fail by itself in the next step.
    for conflict in conflicts:
        new_name = rename[conflict]
        if (new_name not in rename_order) or (
            rename_order.index(conflict) > rename_order.index(new_name)
        ):
            raise ValueError(
                f"Refusing to to rename to existing variable name: {conflict}->{new_name}"
            )

    for old_name in rename_order:
        dataset = dataset.rename({old_name: rename[old_name]})

    return dataset

# Define a function to safely expand dimensions in an xarray dataset,
#  ensures that the data for the new dimensions are in the dataset
def safely_expand_dims(dataset: xr.Dataset, expand_dims: list[str]) -> xr.Dataset:
    """
    Expand dimensions in an xarray dataset, ensuring that the new dimensions are not already in the dataset
    and that the order of dimensions is preserved.
    """
    dims_required = [c for c in dataset.coords if c in expand_dims + list(dataset.dims)]
    dims_missing = [
        (c, i) for i, c in enumerate(dims_required) if c not in dataset.dims
    ]
    dataset = dataset.expand_dims(
        dim=[x[0] for x in dims_missing], axis=[x[1] for x in dims_missing]
    )
    return dataset


out_datasets: dict[str, xr.Dataset] = {}
for out_fname_base, dataset in datasets.items():
    dataset = safely_rename_variable(dataset, post_open_datasets_kwargs.get("rename", {}))

    dataset = safely_expand_dims(dataset, post_open_datasets_kwargs.get("expand_dims", []))

    out_datasets[out_fname_base] = dataset

datasets = out_datasets

Write the datasets to NetCDF

We loop over each dataset in the dictionary and write to netCDF, using the specified options. Note that the compression options must be added to each variable in the dataset.

Code Block
languagepy
In [6]:
# put the engine in the to_netcdf_kwargs, and remove it from the compression options
to_netcdf_kwargs.setdefault("engine", compression_options.pop("engine", "h5netcdf"))
out_nc_files = []
for out_fname_base, dataset in datasets.items():
    to_netcdf_kwargs.update(
        {
            # Add the compression options to the encoding of each variable in the dataset
            "encoding": {var: compression_options for var in dataset},
        }
    )
    out_fname = f"{out_fname_base}.nc"
    dataset.to_netcdf(out_fname, **to_netcdf_kwargs)
    out_nc_files.append(out_fname)
print("NetCDF file(s) created:\n", "\n".join(out_nc_files))
Out [6]:
NetCDF file(s) created:
 data_stream-oper.nc
data_stream-wave.nc

For the latest version please see here: Grib_to_netcdf

Jupyter Viewer
notebookUrlhttps://github.com/ecmwf-projects/dss-notebooks/blob/main/documentation/grib_to_netcdf.ipynb
applicationLinka675ea11-b2c4-336c-bfb6-077e786ef5b2

Summary of changes

This section summarises the changes being made for the various dataset families that are effected. 

ERA5

Legacy converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf era5old {
dimensions:
        longitude = 1440 ;
        latitude = 721 ;
        time = 1 ;
variables:
        float longitude(longitude) ;
                longitude:units = "degrees_east" ;
                longitude:long_name = "longitude" ;
        float latitude(latitude) ;
                latitude:units = "degrees_north" ;
                latitude:long_name = "latitude" ;
        int time(time) ;
                time:units = "hours since 1900-01-01 00:00:00.0" ;
                time:long_name = "time" ;
                time:calendar = "gregorian" ;
        short u10(time, latitude, longitude) ;
                u10:scale_factor = 0.00077681819178887 ;
                u10:add_offset = -0.572623516517769 ;
                u10:_FillValue = -32767s ;
                u10:missing_value = -32767s ;
                u10:units = "m s**-1" ;
                u10:long_name = "10 metre U wind component" ;

// global attributes:
                :Conventions = "CF-1.6" ;
                :history = "2024-09-06 18:44:53 GMT by grib_to_netcdf-2.28.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf -S param -o /cache/data0/adaptor.mars.internal-1725648293.1768346-3616-5-5f732e74-655a-44df-bade-83de1e10126d.nc /cache/tmp/5f732e74-655a-44df-bade-83de1e10126d-adaptor.mars.internal-1725648291.279607-3616-1-tmp.grib" ;
}


Main differences

Panel
borderColor#941331
titleColor#ffffff
borderWidth1
titleBGColor#941331
  • NetCDF3 → NetCDF4 (including compression options)
  • Changes to metadata attributes in files
  • ordering of dimensions
  • changes to time dimension names
  • Splitting of files based on the stepType. The stepType describes how the variable has been aggregated over the time step, and can take values such as instant(=instantaneous) and accum(=accumulated).
  • Format and metadata will be consistent with the post-processed data (e.g. daily statistics)

New converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf era5new {
dimensions:
        valid_time = 1 ;
        latitude = 721 ;
        longitude = 1440 ;
variables:
        int64 number ;
                string number:long_name = "ensemble member numerical id" ;
                string number:units = "1" ;
                string number:standard_name = "realization" ;
        int64 valid_time(valid_time) ;
                string valid_time:long_name = "time" ;
                string valid_time:standard_name = "time" ;
                string valid_time:units = "seconds since 1970-01-01" ;
                string valid_time:calendar = "proleptic_gregorian" ;
        double latitude(latitude) ;
                latitude:_FillValue = NaN ;
                string latitude:units = "degrees_north" ;
                string latitude:standard_name = "latitude" ;
                string latitude:long_name = "latitude" ;
                string latitude:stored_direction = "decreasing" ;
        double longitude(longitude) ;
                longitude:_FillValue = NaN ;
                string longitude:units = "degrees_east" ;
                string longitude:standard_name = "longitude" ;
                string longitude:long_name = "longitude" ;
        string expver ;
        float u10(valid_time, latitude, longitude) ;
                u10:_FillValue = NaNf ;
                u10:GRIB_paramId = 165LL ;
                string u10:GRIB_dataType = "an" ;
                u10:GRIB_numberOfPoints = 1038240LL ;
                string u10:GRIB_typeOfLevel = "surface" ;
                u10:GRIB_stepUnits = 1LL ;
                string u10:GRIB_stepType = "instant" ;
                string u10:GRIB_gridType = "regular_ll" ;
                u10:GRIB_uvRelativeToGrid = 0LL ;
                u10:GRIB_NV = 0LL ;
                u10:GRIB_Nx = 1440LL ;
                u10:GRIB_Ny = 721LL ;
                string u10:GRIB_cfName = "unknown" ;
                string u10:GRIB_cfVarName = "u10" ;
                string u10:GRIB_gridDefinitionDescription = "Latitude/Longitude Grid" ;
                u10:GRIB_iDirectionIncrementInDegrees = 0.25 ;
                u10:GRIB_iScansNegatively = 0LL ;
                u10:GRIB_jDirectionIncrementInDegrees = 0.25 ;
                u10:GRIB_jPointsAreConsecutive = 0LL ;
                u10:GRIB_jScansPositively = 0LL ;
                u10:GRIB_latitudeOfFirstGridPointInDegrees = 90. ;
                u10:GRIB_latitudeOfLastGridPointInDegrees = -90. ;
                u10:GRIB_longitudeOfFirstGridPointInDegrees = -180. ;
                u10:GRIB_longitudeOfLastGridPointInDegrees = 179.75 ;
                u10:GRIB_missingValue = 3.40282346638529e+38 ;
                string u10:GRIB_name = "10 metre U wind component" ;
                string u10:GRIB_shortName = "10u" ;
                u10:GRIB_totalNumber = 0LL ;
                string u10:GRIB_units = "m s**-1" ;
                string u10:long_name = "10 metre U wind component" ;
                string u10:units = "m s**-1" ;
                string u10:standard_name = "unknown" ;
                u10:GRIB_surface = 0. ;
                string u10:coordinates = "number valid_time latitude longitude expver" ;

// global attributes:
                string :GRIB_centre = "ecmf" ;
                string :GRIB_centreDescription = "European Centre for Medium-Range Weather Forecasts" ;
                :GRIB_subCentre = 0LL ;
                string :Conventions = "CF-1.7" ;
                string :institution = "European Centre for Medium-Range Weather Forecasts" ;
                string :history = "2024-09-06T16:33 GRIB to CDM+CF via cfgrib-0.9.14.0/ecCodes-2.36.0 with {\"source\": \"data.grib\", \"filter_by_keys\": {\"stream\": [\"oper\"]}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

ERA5-Land

Legacy converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf era5landold {
dimensions:
        longitude = 3600 ;
        latitude = 1801 ;
        time = 1 ;
variables:
        float longitude(longitude) ;
                longitude:units = "degrees_east" ;
                longitude:long_name = "longitude" ;
        float latitude(latitude) ;
                latitude:units = "degrees_north" ;
                latitude:long_name = "latitude" ;
        int time(time) ;
                time:units = "hours since 1900-01-01 00:00:00.0" ;
                time:long_name = "time" ;
                time:calendar = "gregorian" ;
        short d2m(time, latitude, longitude) ;
                d2m:scale_factor = 0.0014517375787771 ;
                d2m:add_offset = 255.978186484726 ;
                d2m:_FillValue = -32767s ;
                d2m:missing_value = -32767s ;
                d2m:units = "K" ;
                d2m:long_name = "2 metre dewpoint temperature" ;

// global attributes:
                :Conventions = "CF-1.6" ;
                :history = "2024-09-06 19:43:33 GMT by grib_to_netcdf-2.28.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf -S param -o /cache/data0/adaptor.mars.internal-1725651812.3077197-6335-1-ac0ec060-3156-4241-afb4-43a4e1d5ab1a.nc /cache/tmp/ac0ec060-3156-4241-afb4-43a4e1d5ab1a-adaptor.mars.internal-1725651811.1974177-6335-1-tmp.grib" ;
}

Main differences

Panel
borderColor#941331
titleColor#ffffff
borderWidth1
titleBGColor#941331
  • NetCDF3 → NetCDF4 (including compression options)
  • Changes to metadata attributes in files
  • ordering of dimensions
  • Splitting of files based on the stepType. The stepType describes how the variable has been aggregated over the time step, and can take values such as instant(=instantaneous) and accum(=accumulated).
  • Format and metadata will be consistent with the post-processed data (e.g. daily statistics)

New converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf era5landnew {
dimensions:
        valid_time = 1 ;
        latitude = 1801 ;
        longitude = 3600 ;
variables:
        int64 number ;
                string number:long_name = "ensemble member numerical id" ;
                string number:units = "1" ;
                string number:standard_name = "realization" ;
        int64 valid_time(valid_time) ;
                string valid_time:long_name = "time" ;
                string valid_time:standard_name = "time" ;
                string valid_time:units = "seconds since 1970-01-01" ;
                string valid_time:calendar = "proleptic_gregorian" ;
        double latitude(latitude) ;
                latitude:_FillValue = NaN ;
                string latitude:units = "degrees_north" ;
                string latitude:standard_name = "latitude" ;
                string latitude:long_name = "latitude" ;
                string latitude:stored_direction = "decreasing" ;
        double longitude(longitude) ;
                longitude:_FillValue = NaN ;
                string longitude:units = "degrees_east" ;
                string longitude:standard_name = "longitude" ;
                string longitude:long_name = "longitude" ;
        string expver ;
        float d2m(valid_time, latitude, longitude) ;
                d2m:_FillValue = NaNf ;
                d2m:GRIB_paramId = 168LL ;
                string d2m:GRIB_dataType = "fc" ;
                d2m:GRIB_numberOfPoints = 6483600LL ;
                string d2m:GRIB_typeOfLevel = "surface" ;
                d2m:GRIB_stepUnits = 1LL ;
                string d2m:GRIB_stepType = "instant" ;
                string d2m:GRIB_gridType = "regular_ll" ;
                d2m:GRIB_uvRelativeToGrid = 0LL ;
                d2m:GRIB_NV = 0LL ;
                d2m:GRIB_Nx = 3600LL ;
                d2m:GRIB_Ny = 1801LL ;
                string d2m:GRIB_cfName = "unknown" ;
                string d2m:GRIB_cfVarName = "d2m" ;
                string d2m:GRIB_gridDefinitionDescription = "Latitude/Longitude Grid" ;
                d2m:GRIB_iDirectionIncrementInDegrees = 0.1 ;
                d2m:GRIB_iScansNegatively = 0LL ;
                d2m:GRIB_jDirectionIncrementInDegrees = 0.1 ;
                d2m:GRIB_jPointsAreConsecutive = 0LL ;
                d2m:GRIB_jScansPositively = 0LL ;
                d2m:GRIB_latitudeOfFirstGridPointInDegrees = 90. ;
                d2m:GRIB_latitudeOfLastGridPointInDegrees = -90. ;
                d2m:GRIB_longitudeOfFirstGridPointInDegrees = 0. ;
                d2m:GRIB_longitudeOfLastGridPointInDegrees = 359.9 ;
                d2m:GRIB_missingValue = 3.40282346638529e+38 ;
                string d2m:GRIB_name = "2 metre dewpoint temperature" ;
                string d2m:GRIB_shortName = "2d" ;
                d2m:GRIB_totalNumber = 0LL ;
                string d2m:GRIB_units = "K" ;
                string d2m:long_name = "2 metre dewpoint temperature" ;
                string d2m:units = "K" ;
                string d2m:standard_name = "unknown" ;
                d2m:GRIB_surface = 0. ;
                string d2m:coordinates = "number valid_time latitude longitude expver" ;

// global attributes:
                string :GRIB_centre = "ecmf" ;
                string :GRIB_centreDescription = "European Centre for Medium-Range Weather Forecasts" ;
                :GRIB_subCentre = 0LL ;
                string :Conventions = "CF-1.7" ;
                string :institution = "European Centre for Medium-Range Weather Forecasts" ;
                string :history = "2024-09-06T16:34 GRIB to CDM+CF via cfgrib-0.9.14.0/ecCodes-2.36.0 with {\"source\": \"data.grib\", \"filter_by_keys\": {}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

Seasonal Forecasts

Legacy converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf seasonalforecastsold {
dimensions:
        longitude = 360 ;
        latitude = 180 ;
        number = 51 ;
        time = 1 ;
variables:
        float longitude(longitude) ;
                longitude:units = "degrees_east" ;
                longitude:long_name = "longitude" ;
        float latitude(latitude) ;
                latitude:units = "degrees_north" ;
                latitude:long_name = "latitude" ;
        int number(number) ;
                number:long_name = "ensemble_member" ;
        int time(time) ;
                time:units = "hours since 1900-01-01 00:00:00.0" ;
                time:long_name = "time" ;
                time:calendar = "gregorian" ;
        short u10(time, number, latitude, longitude) ;
                u10:scale_factor = 0.000725178414927284 ;
                u10:add_offset = -0.292896463719182 ;
                u10:_FillValue = -32767s ;
                u10:missing_value = -32767s ;
                u10:units = "m s**-1" ;
                u10:long_name = "10 metre U wind component" ;

// global attributes:
                :Conventions = "CF-1.6" ;
                :history = "2024-09-06 17:23:03 GMT by grib_to_netcdf-2.28.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf -S param -o /cache/data3/adaptor.mars.external-1725643382.8773675-11850-19-c9a59396-8d1c-4959-b65c-f1b44cac2259.nc /cache/tmp/c9a59396-8d1c-4959-b65c-f1b44cac2259-adaptor.mars.external-1725643380.1017525-11850-5-tmp.grib" ;
}

Main differences

Panel
borderColor#941331
titleColor#ffffff
borderWidth1
titleBGColor#941331
  • NetCDF3 → NetCDF4 (including compression options)
  • Changes to metadata attributes in files
  • ordering of dimensions
  • changes to time dimension names
  • It is now possible to request data for multiple forecast_reference_times in a single request
  • It is also possible to request multiple models in a single request
  • The metadata details of the time coordinates is no longer over-simplified into a single, one size fits all "time" dimension

New converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf seasonalforecastnew {
dimensions:
        number = 51 ;
        forecast_reference_time = 1 ;
        forecast_period = 1 ;
        latitude = 180 ;
        longitude = 360 ;
variables:
        int64 number(number) ;
                string number:long_name = "ensemble member numerical id" ;
                string number:units = "1" ;
                string number:standard_name = "realization" ;
        int64 forecast_reference_time(forecast_reference_time) ;
                string forecast_reference_time:long_name = "initial time of forecast" ;
                string forecast_reference_time:standard_name = "forecast_reference_time" ;
                string forecast_reference_time:units = "seconds since 1970-01-01" ;
                string forecast_reference_time:calendar = "proleptic_gregorian" ;
        double forecast_period(forecast_period) ;
                forecast_period:_FillValue = NaN ;
                string forecast_period:long_name = "time since forecast_reference_time" ;
                string forecast_period:standard_name = "forecast_period" ;
                string forecast_period:units = "hours" ;
        double latitude(latitude) ;
                latitude:_FillValue = NaN ;
                string latitude:units = "degrees_north" ;
                string latitude:standard_name = "latitude" ;
                string latitude:long_name = "latitude" ;
                string latitude:stored_direction = "decreasing" ;
        double longitude(longitude) ;
                longitude:_FillValue = NaN ;
                string longitude:units = "degrees_east" ;
                string longitude:standard_name = "longitude" ;
                string longitude:long_name = "longitude" ;
        double valid_time ;
                valid_time:_FillValue = NaN ;
                string valid_time:standard_name = "time" ;
                string valid_time:long_name = "time" ;
                string valid_time:units = "seconds since 1970-01-01" ;
                string valid_time:calendar = "proleptic_gregorian" ;
        float u10(number, forecast_reference_time, forecast_period, latitude, longitude) ;
                u10:_FillValue = NaNf ;
                u10:GRIB_paramId = 165LL ;
                string u10:GRIB_dataType = "fc" ;
                u10:GRIB_numberOfPoints = 64800LL ;
                string u10:GRIB_typeOfLevel = "surface" ;
                u10:GRIB_stepUnits = 1LL ;
                string u10:GRIB_stepType = "instant" ;
                string u10:GRIB_gridType = "regular_ll" ;
                u10:GRIB_uvRelativeToGrid = 0LL ;
                u10:GRIB_NV = 0LL ;
                u10:GRIB_Nx = 360LL ;
                u10:GRIB_Ny = 180LL ;
                string u10:GRIB_cfName = "unknown" ;
                string u10:GRIB_cfVarName = "u10" ;
                string u10:GRIB_gridDefinitionDescription = "Latitude/Longitude Grid" ;
                u10:GRIB_iDirectionIncrementInDegrees = 1. ;
                u10:GRIB_iScansNegatively = 0LL ;
                u10:GRIB_jDirectionIncrementInDegrees = 1. ;
                u10:GRIB_jPointsAreConsecutive = 0LL ;
                u10:GRIB_jScansPositively = 0LL ;
                u10:GRIB_latitudeOfFirstGridPointInDegrees = 89.5 ;
                u10:GRIB_latitudeOfLastGridPointInDegrees = -89.5 ;
                u10:GRIB_longitudeOfFirstGridPointInDegrees = 0.5 ;
                u10:GRIB_longitudeOfLastGridPointInDegrees = 359.5 ;
                u10:GRIB_missingValue = 3.40282346638529e+38 ;
                string u10:GRIB_name = "10 metre U wind component" ;
                string u10:GRIB_shortName = "10u" ;
                u10:GRIB_system = 51LL ;
                u10:GRIB_totalNumber = 0LL ;
                string u10:GRIB_units = "m s**-1" ;
                string u10:long_name = "10 metre U wind component" ;
                string u10:units = "m s**-1" ;
                string u10:standard_name = "unknown" ;
                u10:GRIB_surface = 0. ;
                string u10:coordinates = "number time step latitude longitude valid_time" ;

// global attributes:
                :GRIB_edition = 1LL ;
                string :GRIB_centre = "ecmf" ;
                string :GRIB_centreDescription = "European Centre for Medium-Range Weather Forecasts" ;
                :GRIB_subCentre = 0LL ;
                string :Conventions = "CF-1.7" ;
                string :institution = "European Centre for Medium-Range Weather Forecasts" ;
                string :history = "2024-09-06T16:54 GRIB to CDM+CF via cfgrib-0.9.14.0/ecCodes-2.36.0 with {\"source\": \"data.grib\", \"filter_by_keys\": {\"centre\": \"ecmf\"}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

CARRA/CERRA/UERRA

Legacy converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf carraold {
dimensions:
        y = 989 ;
        x = 789 ;
variables:
        int64 time ;
                time:long_name = "initial time of forecast" ;
                time:standard_name = "forecast_reference_time" ;
                time:units = "seconds since 1970-01-01" ;
                time:calendar = "proleptic_gregorian" ;
        double step ;
                step:_FillValue = NaN ;
                step:long_name = "time since forecast_reference_time" ;
                step:standard_name = "forecast_period" ;
                step:units = "hours" ;
        double heightAboveGround ;
                heightAboveGround:_FillValue = NaN ;
                heightAboveGround:long_name = "height above the surface" ;
                heightAboveGround:units = "m" ;
                heightAboveGround:positive = "up" ;
                heightAboveGround:standard_name = "height" ;
        double latitude(y, x) ;
                latitude:_FillValue = NaN ;
                latitude:units = "degrees_north" ;
                latitude:standard_name = "latitude" ;
                latitude:long_name = "latitude" ;
        double longitude(y, x) ;
                longitude:_FillValue = NaN ;
                longitude:units = "degrees_east" ;
                longitude:standard_name = "longitude" ;
                longitude:long_name = "longitude" ;
        double valid_time ;
                valid_time:_FillValue = NaN ;
                valid_time:standard_name = "time" ;
                valid_time:long_name = "time" ;
                valid_time:units = "seconds since 1970-01-01" ;
                valid_time:calendar = "proleptic_gregorian" ;
        float efg10(y, x) ;
                efg10:_FillValue = NaNf ;
                efg10:GRIB_paramId = 260646LL ;
                efg10:GRIB_dataType = "fc" ;
                efg10:GRIB_numberOfPoints = 780321LL ;
                efg10:GRIB_typeOfLevel = "heightAboveGround" ;
                efg10:GRIB_stepUnits = 1LL ;
                efg10:GRIB_stepType = "max" ;
                efg10:GRIB_gridType = "lambert" ;
                efg10:GRIB_DxInMetres = 2500. ;
                efg10:GRIB_DyInMetres = 2500. ;
                efg10:GRIB_LaDInDegrees = 80. ;
                efg10:GRIB_Latin1InDegrees = 80. ;
                efg10:GRIB_Latin2InDegrees = 80. ;
                efg10:GRIB_LoVInDegrees = 326. ;
                efg10:GRIB_NV = 0LL ;
                efg10:GRIB_Nx = 789LL ;
                efg10:GRIB_Ny = 989LL ;
                efg10:GRIB_cfName = "unknown" ;
                efg10:GRIB_cfVarName = "efg10" ;
                efg10:GRIB_gridDefinitionDescription = "Lambert conformal" ;
                efg10:GRIB_iScansNegatively = 0LL ;
                efg10:GRIB_jPointsAreConsecutive = 0LL ;
                efg10:GRIB_jScansPositively = 1LL ;
                efg10:GRIB_latitudeOfFirstGridPointInDegrees = 70.135 ;
                efg10:GRIB_latitudeOfSouthernPoleInDegrees = 0. ;
                efg10:GRIB_longitudeOfFirstGridPointInDegrees = 340.592 ;
                efg10:GRIB_longitudeOfSouthernPoleInDegrees = 0. ;
                efg10:GRIB_missingValue = 9999LL ;
                efg10:GRIB_name = "10 metre eastward wind gust since previous post-processing" ;
                efg10:GRIB_shortName = "10efg" ;
                efg10:GRIB_units = "m s**-1" ;
                efg10:long_name = "10 metre eastward wind gust since previous post-processing" ;
                efg10:units = "m s**-1" ;
                efg10:standard_name = "unknown" ;
                efg10:coordinates = "time step heightAboveGround latitude longitude valid_time" ;

// global attributes:
                :GRIB_edition = 2LL ;
                :GRIB_centre = "enmi" ;
                :GRIB_centreDescription = "Oslo" ;
                :GRIB_subCentre = 255LL ;
                :Conventions = "CF-1.7" ;
                :institution = "Oslo" ;
                :history = "2024-09-06T16:40 GRIB to CDM+CF via cfgrib-0.9.9.1/ecCodes-2.34.1 with {\"source\": \"/cache/tmp/21b28627-bebd-4bda-9a80-776b3885380c-adaptor.mars.external-1725640741.1036253-24617-1-tmp.grib\", \"filter_by_keys\": {}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

Main differences

Panel
borderColor#941331
titleColor#ffffff
borderWidth1
titleBGColor#5dc5e2

  • Compression options

  • Changes coordinate variable names

New converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf carranew {
dimensions:
        valid_time = 1 ;
        y = 989 ;
        x = 789 ;
variables:
        int64 valid_time(valid_time) ;
                string valid_time:long_name = "time" ;
                string valid_time:standard_name = "time" ;
                string valid_time:units = "seconds since 1970-01-01" ;
                string valid_time:calendar = "proleptic_gregorian" ;
        double latitude(y, x) ;
                latitude:_FillValue = NaN ;
                string latitude:units = "degrees_north" ;
                string latitude:standard_name = "latitude" ;
                string latitude:long_name = "latitude" ;
        double longitude(y, x) ;
                longitude:_FillValue = NaN ;
                string longitude:units = "degrees_east" ;
                string longitude:standard_name = "longitude" ;
                string longitude:long_name = "longitude" ;
        string expver ;
        float efg10(valid_time, y, x) ;
                efg10:_FillValue = NaNf ;
                efg10:GRIB_paramId = 260646LL ;
                string efg10:GRIB_dataType = "fc" ;
                efg10:GRIB_numberOfPoints = 780321LL ;
                string efg10:GRIB_typeOfLevel = "heightAboveGround" ;
                efg10:GRIB_stepUnits = 1LL ;
                string efg10:GRIB_stepType = "max" ;
                string efg10:GRIB_gridType = "lambert" ;
                efg10:GRIB_uvRelativeToGrid = 1LL ;
                efg10:GRIB_DxInMetres = 2500. ;
                efg10:GRIB_DyInMetres = 2500. ;
                efg10:GRIB_LaDInDegrees = 80. ;
                efg10:GRIB_Latin1InDegrees = 80. ;
                efg10:GRIB_Latin2InDegrees = 80. ;
                efg10:GRIB_LoVInDegrees = 326. ;
                efg10:GRIB_NV = 0LL ;
                efg10:GRIB_Nx = 789LL ;
                efg10:GRIB_Ny = 989LL ;
                string efg10:GRIB_cfName = "unknown" ;
                string efg10:GRIB_cfVarName = "efg10" ;
                string efg10:GRIB_gridDefinitionDescription = "Lambert conformal" ;
                efg10:GRIB_iScansNegatively = 0LL ;
                efg10:GRIB_jPointsAreConsecutive = 0LL ;
                efg10:GRIB_jScansPositively = 1LL ;
                efg10:GRIB_latitudeOfFirstGridPointInDegrees = 70.135 ;
                efg10:GRIB_latitudeOfSouthernPoleInDegrees = 0. ;
                efg10:GRIB_longitudeOfFirstGridPointInDegrees = 340.592 ;
                efg10:GRIB_longitudeOfSouthernPoleInDegrees = 0. ;
                efg10:GRIB_missingValue = 3.40282346638529e+38 ;
                string efg10:GRIB_name = "10 metre eastward wind gust since previous post-processing" ;
                string efg10:GRIB_shortName = "10efg" ;
                string efg10:GRIB_units = "m s**-1" ;
                string efg10:long_name = "10 metre eastward wind gust since previous post-processing" ;
                string efg10:units = "m s**-1" ;
                string efg10:standard_name = "unknown" ;
                efg10:GRIB_heightAboveGround = 10. ;
                string efg10:coordinates = "valid_time latitude longitude expver" ;

// global attributes:
                string :GRIB_centre = "enmi" ;
                string :GRIB_centreDescription = "Oslo" ;
                :GRIB_subCentre = 255LL ;
                string :Conventions = "CF-1.7" ;
                string :institution = "Oslo" ;
                string :history = "2024-09-06T16:37 GRIB to CDM+CF via cfgrib-0.9.14.0/ecCodes-2.36.0 with {\"source\": \"data.grib\", \"filter_by_keys\": {}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

CAMS global atmospheric composition forecasts

Legacy converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf camsglobalold {
dimensions:
        longitude = 900 ;
        latitude = 451 ;
        level = 3 ;
        time = 2 ;
variables:
        float longitude(longitude) ;
                longitude:units = "degrees_east" ;
                longitude:long_name = "longitude" ;
        float latitude(latitude) ;
                latitude:units = "degrees_north" ;
                latitude:long_name = "latitude" ;
        int level(level) ;
                level:long_name = "model_level_number" ;
        int time(time) ;
                time:units = "hours since 1900-01-01 00:00:00.0" ;
                time:long_name = "time" ;
                time:calendar = "gregorian" ;
        short aermr04(time, level, latitude, longitude) ;
                aermr04:scale_factor = 4.85165910314615e-22 ;
                aermr04:add_offset = 1.33913858161131e-16 ;
                aermr04:_FillValue = -32767s ;
                aermr04:missing_value = -32767s ;
                aermr04:units = "kg kg**-1" ;
                aermr04:long_name = "Dust Aerosol (0.03 - 0.55 um) Mixing Ratio" ;

// global attributes:
                :Conventions = "CF-1.6" ;
                :history = "2024-09-06 16:44:58 GMT by grib_to_netcdf-2.25.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf.bin -S param -o /cache/tmp/b29a7c5c-7228-49d6-a532-d3e3c5ae6f22-adaptor.mars_constrained.internal-1725641098.4198422-8971-22-tmp.nc /cache/tmp/b29a7c5c-7228-49d6-a532-d3e3c5ae6f22-adaptor.mars_constrained.internal-1725641098.34029-8971-21-tmp.grib" ;
}

Main differences

Panel
borderColor#5dc5e2
titleColor#ffffff
borderWidth1
titleBGColor#5dc5e2
  • NetCDF3 → NetCDF4 (including compression options)
  • Changes to metadata attributes in files
  • ordering of dimensions
  • changes to time dimension names
  • Data from multiple forecast_reference_times is archived in a single netCDF file instead of being split
  • The metadata details of the time coordinates is no longer over-simplified into a single, one size fits all "time" dimension
  • pv attribute added to relevant variables

New converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf camsglobalnew {
dimensions:
        forecast_period = 1 ;
        forecast_reference_time = 2 ;
        model_level = 3 ;
        latitude = 451 ;
        longitude = 900 ;
variables:
        double forecast_period(forecast_period) ;
                forecast_period:_FillValue = NaN ;
                string forecast_period:long_name = "time since forecast_reference_time" ;
                string forecast_period:standard_name = "forecast_period" ;
                string forecast_period:units = "hours" ;
        int64 forecast_reference_time(forecast_reference_time) ;
                string forecast_reference_time:long_name = "initial time of forecast" ;
                string forecast_reference_time:standard_name = "forecast_reference_time" ;
                string forecast_reference_time:units = "seconds since 1970-01-01" ;
                string forecast_reference_time:calendar = "proleptic_gregorian" ;
        double model_level(model_level) ;
                model_level:_FillValue = NaN ;
                string model_level:long_name = "hybrid level" ;
                string model_level:units = "1" ;
                string model_level:positive = "down" ;
                string model_level:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
        double latitude(latitude) ;
                latitude:_FillValue = NaN ;
                string latitude:units = "degrees_north" ;
                string latitude:standard_name = "latitude" ;
                string latitude:long_name = "latitude" ;
                string latitude:stored_direction = "decreasing" ;
        double longitude(longitude) ;
                longitude:_FillValue = NaN ;
                string longitude:units = "degrees_east" ;
                string longitude:standard_name = "longitude" ;
                string longitude:long_name = "longitude" ;
        double valid_time(forecast_reference_time, forecast_period) ;
                valid_time:_FillValue = NaN ;
                string valid_time:standard_name = "time" ;
                string valid_time:long_name = "time" ;
                string valid_time:units = "seconds since 1970-01-01T00:00:00" ;
                string valid_time:calendar = "proleptic_gregorian" ;
        float aermr04(forecast_period, forecast_reference_time, model_level, latitude, longitude) ;
                aermr04:_FillValue = NaNf ;
                aermr04:GRIB_paramId = 210004LL ;
                string aermr04:GRIB_dataType = "fc" ;
                aermr04:GRIB_numberOfPoints = 405900LL ;
                string aermr04:GRIB_typeOfLevel = "hybrid" ;
                aermr04:GRIB_stepUnits = 1LL ;
                string aermr04:GRIB_stepType = "instant" ;
                string aermr04:GRIB_gridType = "regular_ll" ;
                aermr04:GRIB_uvRelativeToGrid = 0LL ;
                aermr04:GRIB_NV = 276LL ;
                aermr04:GRIB_Nx = 900LL ;
                aermr04:GRIB_Ny = 451LL ;
                string aermr04:GRIB_cfName = "unknown" ;
                string aermr04:GRIB_cfVarName = "aermr04" ;
                string aermr04:GRIB_gridDefinitionDescription = "Latitude/longitude" ;
                aermr04:GRIB_iDirectionIncrementInDegrees = 0.4 ;
                aermr04:GRIB_iScansNegatively = 0LL ;
                aermr04:GRIB_jDirectionIncrementInDegrees = 0.4 ;
                aermr04:GRIB_jPointsAreConsecutive = 0LL ;
                aermr04:GRIB_jScansPositively = 0LL ;
                aermr04:GRIB_latitudeOfFirstGridPointInDegrees = 90. ;
                aermr04:GRIB_latitudeOfLastGridPointInDegrees = -90. ;
                aermr04:GRIB_longitudeOfFirstGridPointInDegrees = 0. ;
                aermr04:GRIB_longitudeOfLastGridPointInDegrees = 359.6 ;
                aermr04:GRIB_missingValue = 3.40282346638529e+38 ;
                string aermr04:GRIB_name = "Dust Aerosol (0.03 - 0.55 um) Mixing Ratio" ;
                aermr04:GRIB_pv = 0., 2.0003650188446, 3.10224103927612, 4.66608381271362, 6.82797718048096, 9.74696636199951, 13.6054239273071, 18.6089305877686, 24.9857177734375, 32.985710144043, 42.8792419433594, 54.9554634094238, 69.5205764770508, 86.895881652832, 107.415740966797, 131.425506591797, 159.279403686523, 191.338562011719, 227.968948364258, 269.539581298828, 316.420745849609, 368.982360839844, 427.592498779297, 492.616027832031, 564.413452148438, 643.339904785156, 729.744140625, 823.967834472656, 926.344909667969, 1037.201171875, 1156.85363769531, 1285.6103515625, 1423.77014160156, 1571.62292480469, 1729.44897460938, 1897.51928710938, 2076.09594726562, 2265.431640625, 2465.7705078125, 2677.34814453125, 2900.39135742188, 3135.11938476562, 3381.74365234375, 3640.46826171875, 3911.49047851562, 4194.9306640625, 4490.8173828125, 4799.1494140625, 5119.89501953125, 5452.99072265625, 5798.3447265625, 6156.07421875, 6526.94677734375, 6911.87060546875, 7311.869140625, 7727.412109375, 8159.35400390625, 8608.525390625, 9076.400390625, 9562.6826171875, 10065.978515625, 10584.6318359375, 11116.662109375, 11660.0673828125, 12211.5478515625, 12766.873046875, 13324.6689453125, 13881.3310546875, 14432.1396484375, 14975.615234375, 15508.2568359375, 16026.115234375, 16527.322265625, 17008.7890625, 17467.61328125, 17901.62109375, 18308.43359375, 18685.71875, 19031.2890625, 19343.51171875, 19620.04296875, 19859.390625, 20059.931640625, 20219.6640625, 20337.86328125, 20412.30859375, 20442.078125, 20425.71875, 20361.81640625, 20249.51171875, 20087.0859375, 19874.025390625, 19608.572265625, 19290.2265625, 18917.4609375, 18489.70703125, 18006.92578125, 17471.83984375, 16888.6875, 16262.046875, 15596.6953125, 14898.453125, 14173.32421875, 13427.76953125, 12668.2578125, 11901.33984375, 11133.3046875, 10370.17578125, 9617.515625, 8880.453125, 8163.375, 7470.34375, 6804.421875, 6168.53125, 5564.3828125, 4993.796875, 4457.375, 3955.9609375, 3489.234375, 3057.265625, 2659.140625, 2294.2421875, 1961.5, 1659.4765625, 1387.546875, 1143.25, 926.5078125, 734.9921875, 568.0625, 424.4140625, 302.4765625, 202.484375, 122.1015625, 62.78125, 22.8359375, 3.75781297683716, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 3.81999996079685e-08, 6.76070021654596e-06, 2.43480008066399e-05, 5.89219998801127e-05, 0.000111914298031479, 0.00019857739971485, 0.000340379687258974, 0.000561555323656648, 0.000889697927050292, 0.00135280552785844, 0.00199183798395097, 0.00285712420009077, 0.00397095363587141, 0.00537781463935971, 0.00713337678462267, 0.00926146004348993, 0.0118060223758221, 0.014815628528595, 0.0183184519410133, 0.0223548449575901, 0.0269635207951069, 0.0321760959923267, 0.03802639991045, 0.0445479601621628, 0.051773015409708, 0.0597284138202667, 0.0684482529759407, 0.077958308160305, 0.0882857367396355, 0.099461667239666, 0.111504651606083, 0.124448128044605, 0.138312891125679, 0.153125032782555, 0.168910413980484, 0.185689449310303, 0.2034912109375, 0.222332864999771, 0.242244005203247, 0.26324188709259, 0.285354018211365, 0.308598458766937, 0.332939088344574, 0.358254194259644, 0.384363323450089, 0.411124765872955, 0.438391208648682, 0.46600329875946, 0.493800312280655, 0.521619200706482, 0.549301147460938, 0.576692163944244, 0.603648066520691, 0.630035817623138, 0.655735969543457, 0.680643022060394, 0.704668998718262, 0.727738738059998, 0.749796569347382, 0.770797550678253, 0.790716767311096, 0.809536039829254, 0.827256083488464, 0.843881130218506, 0.859431803226471, 0.873929262161255, 0.887407541275024, 0.899900496006012, 0.911448180675507, 0.922095656394958, 0.9318807721138, 0.94085955619812, 0.949064433574677, 0.956549525260925, 0.963351726531982, 0.969513416290283, 0.975078403949738, 0.980071604251862, 0.984541893005371, 0.988499522209167, 0.991984009742737, 0.995002508163452, 0.99763011932373, 1. ;
                string aermr04:GRIB_shortName = "aermr04" ;
                string aermr04:GRIB_units = "kg kg**-1" ;
                string aermr04:long_name = "Dust Aerosol (0.03 - 0.55 um) Mixing Ratio" ;
                string aermr04:units = "kg kg**-1" ;
                string aermr04:standard_name = "unknown" ;
                string aermr04:coordinates = "step time hybrid latitude longitude valid_time" ;

// global attributes:
                string :GRIB_centre = "ecmf" ;
                string :GRIB_centreDescription = "European Centre for Medium-Range Weather Forecasts" ;
                :GRIB_subCentre = 0LL ;
                string :Conventions = "CF-1.7" ;
                string :institution = "European Centre for Medium-Range Weather Forecasts" ;
                string :history = "2024-09-06T16:46 GRIB to CDM+CF via cfgrib-0.9.14.0/ecCodes-2.36.0 with {\"source\": \"data.grib\", \"filter_by_keys\": {\"typeOfLevel\": \"hybrid\"}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

CAMS global reanalysis (EAC4)

Legacy converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf camseac4old {
dimensions:
        longitude = 480 ;
        latitude = 241 ;
        level = 2 ;
        time = 1 ;
variables:
        float longitude(longitude) ;
                longitude:units = "degrees_east" ;
                longitude:long_name = "longitude" ;
        float latitude(latitude) ;
                latitude:units = "degrees_north" ;
                latitude:long_name = "latitude" ;
        int level(level) ;
                level:units = "millibars" ;
                level:long_name = "pressure_level" ;
        int time(time) ;
                time:units = "hours since 1900-01-01 00:00:00.0" ;
                time:long_name = "time" ;
                time:calendar = "gregorian" ;
        short c2h6(time, level, latitude, longitude) ;
                c2h6:scale_factor = 1.6469724757173e-17 ;
                c2h6:add_offset = 5.52540709711356e-13 ;
                c2h6:_FillValue = -32767s ;
                c2h6:missing_value = -32767s ;
                c2h6:units = "kg kg**-1" ;
                c2h6:long_name = "Ethane" ;
                c2h6:standard_name = "mass_fraction_of_ethane_in_air" ;

// global attributes:
                :Conventions = "CF-1.6" ;
                :history = "2024-09-06 16:46:49 GMT by grib_to_netcdf-2.25.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf.bin -S param -o /cache/data6/adaptor.mars.internal-1725641209.203058-26900-7-448cb5be-9939-4e25-a58a-1d06dd2c856f.nc /cache/tmp/448cb5be-9939-4e25-a58a-1d06dd2c856f-adaptor.mars.internal-1725641208.8677917-26900-12-tmp.grib" ;
}

Main differences

Panel
borderColor#5dc5e2
titleColor#ffffff
borderWidth1
titleBGColor#5dc5e2
  • NetCDF3 → NetCDF4 (including compression options)
  • Changes to metadata attributes in files
  • ordering of dimensions
  • changes to time dimension names
  • pv attribute added to relevant variables

New converter

Code Block
languagepy
titleNetCDF header
collapsetrue
netcdf camseac4new {
dimensions:
        valid_time = 1 ;
        pressure_level = 2 ;
        latitude = 241 ;
        longitude = 480 ;
variables:
        int64 valid_time(valid_time) ;
                string valid_time:long_name = "time" ;
                string valid_time:standard_name = "time" ;
                string valid_time:units = "seconds since 1970-01-01" ;
                string valid_time:calendar = "proleptic_gregorian" ;
        double pressure_level(pressure_level) ;
                pressure_level:_FillValue = NaN ;
                string pressure_level:long_name = "pressure" ;
                string pressure_level:units = "hPa" ;
                string pressure_level:positive = "down" ;
                string pressure_level:stored_direction = "decreasing" ;
                string pressure_level:standard_name = "air_pressure" ;
        double latitude(latitude) ;
                latitude:_FillValue = NaN ;
                string latitude:units = "degrees_north" ;
                string latitude:standard_name = "latitude" ;
                string latitude:long_name = "latitude" ;
                string latitude:stored_direction = "decreasing" ;
        double longitude(longitude) ;
                longitude:_FillValue = NaN ;
                string longitude:units = "degrees_east" ;
                string longitude:standard_name = "longitude" ;
                string longitude:long_name = "longitude" ;
        float c2h6(valid_time, pressure_level, latitude, longitude) ;
                c2h6:_FillValue = NaNf ;
                c2h6:GRIB_paramId = 217045LL ;
                string c2h6:GRIB_dataType = "an" ;
                c2h6:GRIB_numberOfPoints = 115680LL ;
                string c2h6:GRIB_typeOfLevel = "isobaricInhPa" ;
                c2h6:GRIB_stepUnits = 1LL ;
                string c2h6:GRIB_stepType = "instant" ;
                string c2h6:GRIB_gridType = "regular_ll" ;
                c2h6:GRIB_uvRelativeToGrid = 0LL ;
                c2h6:GRIB_NV = 0LL ;
                c2h6:GRIB_Nx = 480LL ;
                c2h6:GRIB_Ny = 241LL ;
                string c2h6:GRIB_cfName = "mass_fraction_of_ethane_in_air" ;
                string c2h6:GRIB_cfVarName = "c2h6" ;
                string c2h6:GRIB_gridDefinitionDescription = "Latitude/Longitude Grid" ;
                c2h6:GRIB_iDirectionIncrementInDegrees = 0.75 ;
                c2h6:GRIB_iScansNegatively = 0LL ;
                c2h6:GRIB_jDirectionIncrementInDegrees = 0.75 ;
                c2h6:GRIB_jPointsAreConsecutive = 0LL ;
                c2h6:GRIB_jScansPositively = 0LL ;
                c2h6:GRIB_latitudeOfFirstGridPointInDegrees = 90. ;
                c2h6:GRIB_latitudeOfLastGridPointInDegrees = -90. ;
                c2h6:GRIB_longitudeOfFirstGridPointInDegrees = 0. ;
                c2h6:GRIB_longitudeOfLastGridPointInDegrees = 359.25 ;
                c2h6:GRIB_missingValue = 3.40282346638529e+38 ;
                string c2h6:GRIB_name = "Ethane" ;
                string c2h6:GRIB_shortName = "c2h6" ;
                c2h6:GRIB_totalNumber = 0LL ;
                string c2h6:GRIB_units = "kg kg**-1" ;
                string c2h6:long_name = "Ethane" ;
                string c2h6:units = "kg kg**-1" ;
                string c2h6:standard_name = "mass_fraction_of_ethane_in_air" ;
                c2h6:GRIB_number = 0LL ;
                string c2h6:coordinates = "valid_time isobaricInhPa latitude longitude" ;

// global attributes:
                string :GRIB_centre = "ecmf" ;
                string :GRIB_centreDescription = "European Centre for Medium-Range Weather Forecasts" ;
                :GRIB_subCentre = 0LL ;
                string :Conventions = "CF-1.7" ;
                string :institution = "European Centre for Medium-Range Weather Forecasts" ;
                string :history = "2024-09-06T16:48 GRIB to CDM+CF via cfgrib-0.9.14.0/ecCodes-2.36.0 with {\"source\": \"data.grib\", \"filter_by_keys\": {\"typeOfLevel\": \"isobaricInhPa\"}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

New Coordinate variable names

Some of the coordinate variables have been rename to address issues in the legacy netCDF conversion regarding non CF complaint variables, and unclear and overlapping definitions of the time-dimensions.

Table 1: New Coordinate variable names

Coordinate variable name in new netCDF conversionCoordinate variable name in legacy netCDF conversion for regional reanalysis (CARRA/CERRA/UERRA)Coordinate variable name in legacy netCDF conversion for global reanalysis (ERA5 family), seasonal forecasts and CAMS datasetsGRIB/MARS key(s)Any other business
latitudelatitudelatitudelatitude
longitudelongitudelongitudelongitude
valid_timevalid_timetimevalidityDate+ validityTime
  • Time dimension for "Reanalysis" data
  • A coordinate for "Forecast" data
forecast_reference_timetimetimetimeA time dimension for "forecast" data.
forecast_periodsteptimestepA time dimension for "forecast" data.
indexing_timeindexing_timetimeindexingDate + indexingTime

This coordinate is only used in 'seasonal-monthly-*-levels' and 'seasonal-postprocessed-*' datasets.

This accounts for data where the forecast_reference_time differs for each ensemble member, specifically this happens when the ensemble members are initialised following a lagged start approach. In those situations instead of forecast_reference_time a "nominal start date" is used encoded with this name of "indexing_time".

forecastMonthforecastMonthtimeforecastMonth

This coordinate is only used in 'seasonal-monthly-*-levels' and 'seasonal-postprocessed-*' datasets.

This is the number of step w.r.t the forecast_reference_time or indexing_time.

The convention used for the numbering is "1" is the first complete month after the nominal start date (forecast_reference_time/indexing_time), for instance if the nominal start date is 1st November, forecastMonth=1 means November.

valid_month

(this may be renamed valid_time in future editions)

valid_monthtimemonthlyVerificationDate + validityTime

This is required by some monthly datasets where the "valid_time" differs between start of month and end of month, depending on variable.

valid_month not used for ERA5 family and CAMS datasets.

pressure_levelisobaricInhPalevel

levelist + levtype or level + typeOfLevel


model_levelhybridlevellevelist + levtype or level + typeOfLevel
numbernumber numbernumber

Legacy converter

Note

To assist in transition, the legacy convertor will be made available programatically via the cdsapi, however this is considered a deprecated format and is no longer supported. To use the legacy convertor, you should update you cdsapi request with:

 "data_format": "netcdf_legacy" 

Please be aware that the resulting files are netCDF3.


Info
iconfalse

This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS) and Copernicus Climate Change Service (C3S).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS and C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014 and Contribution Agreement signed on 22/07/2021). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.

The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.

Related articles

Content by Label
max3
cqllabel in ("cams","c3s","cds","netcdf","grib")