Good morning,
I'm running cy43r3 and I would like to process the ICMUA output files. The new files contain a mix of grib1 and grib2 fields that is not easly processable by the cdo provided our computing center:
cdo -R -t ecmwf -f nc copy -remapbil,r360x180 ICMUA4317+000000 ICMUA4317+000000.nc cdo(2) remapbil: Process started Segmentation fault (core dumped) grib_ls ICMUA4317+000000 ICMUA4317+000000 edition centre typeOfLevel level dataDate stepRange dataType shortName packingType gridType 1 ecmf isobaricInhPa 1000 20160925 0 fc q grid_simple reduced_gg 1 ecmf isobaricInhPa 925 20160925 0 fc q grid_simple reduced_gg 1 ecmf isobaricInhPa 850 20160925 0 fc q grid_simple reduced_gg 1 ecmf isobaricInhPa 700 20160925 0 fc q grid_simple reduced_gg . . . 2 ecmf hybrid 57 20160925 0 fc vtpha grid_simple reduced_gg 2 ecmf hybrid 58 20160925 0 fc vtpha grid_simple reduced_gg 2 ecmf hybrid 59 20160925 0 fc vtpha grid_simple reduced_gg 2 ecmf hybrid 60 20160925 0 fc vtpha grid_simple reduced_gg 360 of 360 grib messages in ICMUA4317+000000_grib2 360 of 360 total grib messages in 1 files
I tried to split the file into grib1 and grib2 fields. The grib1 file looks fine and can be processed by cdo, but the grib2 file has the gridSize changed (maybe?) which seems to prevent further post processing:
grib_copy ICMUA4317+000000 -w edition=1 ICMUA4317+000000_grib1 grib_copy ICMUA4317+000000 -w edition=2 ICMUA4317+000000_grib2 grib_ls ICMUA4317+000000_grib2 ICMUA4317+000000_grib2 edition centre date dataType gridType stepRange typeOfLevel level shortName packingType 2 ecmf 20160925 fc reduced_gg 0 hybrid 1 q grid_simple 2 ecmf 20160925 fc reduced_gg 0 hybrid 2 q grid_simple 2 ecmf 20160925 fc reduced_gg 0 hybrid 3 q grid_simple 2 ecmf 20160925 fc reduced_gg 0 hybrid 4 q grid_simple . . . cdo -R -t ecmwf -f nc copy -remapbil,r360x180 ICMUA4317+000000_grib2 ICMUA4317+000000.nc cdo(2) remapbil: Process started Error (gribapiGetGridRegular): numberOfPoints (35718) and gridSize (343597383520) differ!
I tried to convert the mixed files with grib_to_netcdf but here I get an error related to the gridType:
grib_to_netcdf -T -o ICMUA4317+000000.nc ICMUA4317+000000 grib_to_netcdf: Version 2.10.0 grib_to_netcdf: Processing input file 'ICMUA4317+000000'. grib_to_netcdf: Found 489 GRIB fields in 1 file. grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate grib_to_netcdf: Creating netCDF file 'ICMUA4317+000000.nc' grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $ grib_to_netcdf: Creating large (64 bit) file format. ECCODES ERROR : First GRIB is not on a regular lat/lon grid or on a regular Gaussian grid. Exiting.
How to I get my hands on the sweet data inside this file? 
Cheers, Jan
4 Comments
Unknown User (jstreffi)
Problem solved. For posterity: In the namelist that came with the input files, 3D output on both pressure and surface levels was turned on. Pressure level output is written in grib1 and surface level output written in grib2. Resulting in the unprocessable files I described. I just turned off the surface level output.
I also deactivated the K-Index(260121) and Total totals index (260123) that were written as grib2 into the 2D output file.
Cheers, Jan
Unknown User (nagc)
Hi Jan,
Sorry, I missed this page earlier.
You are a bit ahead of the curve, but my understanding was that the ICM*UA+* output file for 43r3 was just a rename of the 40r1 ICM*GG+* file, so the contents should be the same and can be treated the same way, though some of the grib keys might have changed (for the new cubic grid for example).
I think it's the other way around, the surface fields are GRIB1 and the pressure level output is GRIB2. Only GRIB2 can encode greater than 127 levels so it must be that way around.
I know that cdo can choke with there are multiple level types involved. The surface fields include the sub-surface levels and mixing this with the pressure level fields in one file seems to confuse cdo.
The recommendation is to split the level types first (see How to convert GRIB to netCDF for much more detail. You can split the vertical axes in two ways. Either using cdo:
or by using the grib_copy command that comes with ecCodes (which personally I prefer):
where 'typeOfLevel' is a grib key.
Once the level types are split cdo should work ok?
Was there another problem with the K-index and Total totals index?
Cheers, Glenn
Unknown User (jstreffi)
Hello Glenn,
that is an interesting point. I removed the lines
and kept
This should leave me with pressure level output if I interpret the names correctly.
grib_lstells me that the ICMSH output contain fields withtypeOfLevel=isobaricInhPaandedition=1. I interpreted that as grib1 pressure level data and the contents do look like pressure data. Something wrong here? K-index and Total totals index were also ofedition=2and resulted in inability to post-process the ICMGG file until I removed them from the output.Ultimately I don't have much use for surface level data, so I am happy not writing it out. For me it would only make the model slower and use up disc space anyway.
Cheers, Jan
Unknown User (de3j)
Hi Jan
I never had this problem with 43r3 using our post processing in esm-tools. I do what Glenn suggests: split into one file per vertical level and also GG and SH. So I get output files like ending with:
SH_isobaricInhPa (U,V,T,Z)
GG_isobaricInhPa (Q,O3)
SH_hybrid
GG_hybrid
GG_surface
but with 43r3 I also get "UA_isobaricInhPa" and "UA_hybrid" rather than "GG_isobaricInhPa" and "GG_hybrid".
I do the following:
cat ICMUA* > ICMUACAT
grib_copy -w typeOfLevel=isobaricInhPa ICMUACAT ICMUACAT_isobaricInhPa
cdo --eccodes -f nc4c -z zip_4 -t ecmwf -setgridtype,regular ICMUACAT_isobaricInhPa h6mv_19820101_19820105_UA_isobaricInhPa.nc
If CDO has been compiled with ecCodes, the above will convert the UA output to a netCDF file on a regular grid with moderate compression.
Cheers
Joakim
PS: The error you get is however exactly what I get when I try to post process the output from WAM at T511. WAM (for some reason) writes data to different grid types (regular,reduced etc) depending on the resolution. No idea why...