Page tree
Skip to end of metadata
Go to start of metadata

Available data

The CAMS Regional service provides daily near-real-time (NRT) analyses and forecasts. 

Data is delivered via different platforms, and in different data formats:

  • Online data. This is analyses and forecast data from the past 30 days. This data is stored at the CAMS regional DCPC data centre operated at Météo-France. This dataset comes in different formats, depending on the model:
    • Data from the ENSEMBLE is provided in GRIB2 and NetCDF format
    • Data from individual models is provided in NetCDF format only
  • Archive data. The online data described above is after 30 days moved to a data archive. This archive data is stored at the BDAP data archive at Météo-France.
    • The archive data is available in GRIB2 format only

Problem

The GRIB2 format used for the CAMS Regional data is specific to Météo-France, hence decoding the GRIB data with non-Météo-France tools can fail or yield unexpected results. For example, using grib_to_netcdf from the ECMWF ecCodes tools:

me@machine:~> grib_to_netcdf -o file.nc W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+ALLLEVELS+CO+0H24H_C_LFPW_20160301000000.grib2
grib_to_netcdf: Version 2.5.0
grib_to_netcdf: Processing input file 'W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+ALLLEVELS+CO+0H24H_C_LFPW_20160301000000.grib2'.
ECCODES ERROR   :  Wrong number of fields
ECCODES ERROR   :  File contains 200 GRIBs, 200 left in internal description, 25 in request
ECCODES ERROR   :  The fields are not considered distinct!

ECCODES ERROR   :  Hint: This may be due to several fields having the same validity time.
ECCODES ERROR   :  Try using the -T option (Do not use time of validity)

(In this case the conversion fails because the GRIB2 data contains non-unique Date/Time/Step combinations, which in NetCDF is not allowed.)

This is only an example, other procedures with the CAMS Regional GRIB2 data might yield other errors or unexpected results.

Solution

Most CAMS Regional data is available in GRIB2 and NetCDF format, so use the data in NetCDF format where available. Only the archive data is not available in NetCDF, so this data you have to convert yourself from GRIB to NetCDF, using the procedure below.

The procedure described here is specific to GRIB2 data from Météo-France. It should not be expected to work with GRIB2 data from other sources.

Prerequisites

You need:

  • Linux
  • ecCodes or GRIB_API
  • CDO
  • NCO (commands ncrename, ncatted, ncpdq)
  • NCView (optional, to visualise result)
  • Panoply (optional, to visualise result)

The conversion process has been tested with the following versions of tools on these operating systems (OS):

  • Mageia 6 : Linux 4.8.2-desktop-1.mga6
    • GRIB_API v1.19.0
    • CDO v1.7.2
    • NCO v4.5.5
    • NCView v2.1.2
    • Panoply v4.4.3
  • CentOS 6.5 : Linux 2.6.32-431.29.2.el6.x86_64
    • GRIB_API v1.19.16
    • CDO v1.6.1rc6
    • NCO v4.3.6
    • NCView v2.1.2
    • Panoply v4.4.3
  • OpenSuse : Linux 4.4.73-18.17-default
    • ecCodes v2.5.0
    • CDO v1.7.2
    • NCO v4.4.6
    • NCView v2.1.2
    • Panoply v4.0
How to display versions of tools and OS used
# uname -sr
# grib_ls -V
# cdo --version (or cdo -V)
# ncrename --version
# ncview -version
# panoply --version

About units

The CAMS Regional production data is downloadable in these formats and units:


download in GRIB2 formatdownload in NetCDF format
Chemical specieskg/m3 *μg/m3
Pollenparticles/m3particles/m3

* The procedure below describes a format conversion. It does not modify the data values or units. Hence, when you convert GRIB2 files of chemical species (in kg/m3), the resulting NetCDF file will also be in kg/m3.

GRIB2 to NetCDF conversion

The following steps are to be executed in the given order.

The NetCDF files produced with this procedure are not byte-identical with the equivalent NetCDF files downloadable from the CAMS Regional service, due to a slightly different way how metadata is handled.

Step 1: Get metadata from the GRIB2 file

We read some metadata from the GRIB2 file, namely "species code", "parameter number" and "date":

  # grib_get -p constituentType,parameterNumber,date camsdatafile.grib2 | head -n 1

Example:

  # grib_get -p constituentType,parameterNumber,date W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.grib2 | head -n 1
  40008 0 20180215

In this example we identified: "species code (constituentType)" = 40008, "parameterNumber" = 0, "date" = 20180215. Keep note of these values, we will need them later.

Step 2: Look up metadata

For the GRIB2 "species code" and "parameterNumber", look up the corresponding "short name" and "long name" in the following table and note them for later:

"species code (constituentType)" from GRIB2,

indicates chemical constituent

"parameterNumber" from GRIB2

indicates units

short name

long name

units
0

0


O3Ozonekg.m^-3
50
NO2Nitrogen Dioxidekg.m^-3
8

0


SO2Sulphur Dioxidekg.m^-3
40
COCarbon monoxidekg.m^-3
400080
PM10PM10 aerosolkg.m^-3
400090
PM2.5PM2.5 aerosolkg.m^-3
110
NONitrogen monoxidekg.m^-3
90
NH3Ammoniakg.m^-3
600130
NMVOCNon Methane Volatile Organic compounds expressed as Carbonkg.m^-3
60018

0


PANsFamily of PeroxyAcetyl Nitratekg.m^-3
6400059
BirchPollenBirch Pollen particlesparticles.m^-3
6400159
GrassPollenGrass Pollen particlesparticles.m^-3
6400259
OlivePollenOlive Pollen particlesparticles.m^-3

Source: GRIB2 Format (cf. DOC WMO N°306 FM92 : "GRIB(edition 2) Codes and Flags"): https://codes.wmo.int/grib2
Template used: Version 4.40.
Table Version: Version 5
Compression used: "grid_jpeg"
Discipline: 0 (Meteorological products)
Category: 20 (Atmospheric Chemical Constituent)

The tables contains standard WMO codes, plus local codes for the following species because standard codes have not been defined: PM10 (40008), PM25 (40009), NMVOC (60013), PANS (60018) and pollen (64000, 64001, 64002).

Step 3: Transform to "height" type

We transforms all levels in the GRIB2 file to "height" type to avoid the ground level being misinterpreted.

  # grib_set -s typeOfLevel=heightAboveGround camsdatafile.grib2 camsdatafile.t.h.grib2

Example:

  # grib_set -s typeOfLevel=heightAboveGround \
W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.grib2 \
W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.h.grib

Step 4: Reorder the GRIB messages by time and elevation ascending order

For FORECAST:

  # grib_copy -B "stepRange:i asc, level:i asc" camsdatafile.t.h.grib2 camsdatafile.t.o.grib2

For ANALYSIS:

  # grib_copy -B "dataTime:i asc, level:i asc" camsdatafile.t.h.grib2 camsdatafile.t.o.grib2

Step 5: Convert GRIB2 to NetCDF

We do the actual format conversion from GRIB2 to NetCDF, using cdo:

  # cdo -z zip1 -f nc4 copy camsdatafile.t.o.grib2 camsdatafile.t.nc

Example:

  # cdo -z zip1 -f nc4 copy \
W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.o.grib \ W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.nc cdo copy: Processed 7000000 values from 1 variable over 25 timesteps ( 3.33s )

Step 6: Rename variable

cdo gave the variables a non explicit name, e.g. "name" = "param0.20.0". Hence we replace the variable name with the correct "short name", as identified in step 2.

For parameters with "parameterNumber" = 0:

  # ncrename -v param0.20.0,"short name" camsdatafile.t.nc camsdatafile.t.rn.nc

For parameters with "parameterNumber" =59:

  # ncrename -v param59.20.0,"short name" camsdatafile.t.nc camsdatafile.t.rn.nc

Example:

  # ncrename -v param0.20.0,PM10 \
W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.nc \
W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.rn.nc
  ncrename: In total renamed 0 attributes, 0 dimensions, 0 groups, and 1 variable

Step 7: Add metadata for name and units

To comply with CF Conventions we add metadata for "long name" and units, as identified in step 2.

For parameters with "parameterNumber" = 0:

  # ncatted -a long_name,"short name",c,c,"long name" -a units,"short name",c,c,"kg.m^-3" camsdatafile.t.rn.nc

For parameters with "parameterNumber" = 59:

  # ncatted -a long_name,"short name",c,c,"long name" -a units,"short name",c,c,"particles.m^-3" camsdatafile.t.rn.nc

Example:

  # ncatted -a long_name,PM10,c,c,"PM10 aerosol" -a units,PM10,c,c,"kg.m^-3" \
W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.rn.nc

Step 8: Add metadata for "date"

We add metadata for the "date" dimension:

For FORECAST: 

  # ncatted -a long_name,time,c,c,"FORECAST time from YYYYMMDD" camsdatafile.t.rn.nc

For ANALYSIS:

  # ncatted -a long_name,time,c,c,"ANALYSIS time from YYYYMMDD" camsdatafile.t.rn.nc

Example:

  # ncatted -a long_name,time,c,c,"FORECAST time from 20180215" \
  W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.rn.nc

Step 9: Reverse latitude

We reverse the latitude dimension to order the data from South to North:

  # ncpdq -a '-lat' camsdatafile.t.rn.nc camsdatafile.nc

Example:

  # ncpdq -a '-lat' \
  W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.rn.nc \
  W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.nc

Step 10: Delete temporary files

  # rm camsdatafile.t.*

Example:

  # rm W_fr-meteofrance,MODEL,ENSEMBLE+FORECAST+SURFACE+PM10+0H24H_C_LFPW_20180215000000.t.*