How to merge multiple GRIB or NetCDF files

For GRIB:

On Linux you can use the grib_copy tool. This tools comes with ECMWF's ecCodes package:

grib_copy grib_file_1 grib_file_2 ... output_grib_file

On Linux you can simply concatenate the GRIB files:

cat grib_file_1 grib_file_2 ... grib_file_N > output_grib_file

For NetCDF (unsupported by ECMWF)

Care needs to be taken when merging NetCDF files, as differences in the structure of the files can cause issue. Other issues such as the total size of the merged files also needs to be considered.

If the files are of similar structure, one way to merge netcdf files is using CDO merge:

cdo merge infile1 infile2 infile3 outfile

or if a valid time'

cdo mergetime infile1 infile2 infile3 infile4 outfile

Another option could be NCO (NetCDF Operators). However, please be sure that the files are 'unpacked' before merging them. The NCO 'ncpdq' command can be used to produce an unpacked version of the files.

Here an example of merging for ERA5 NetCDF files from the CDS:

Download the data:

ERA5 CDS API request

import cdsapi
c = cdsapi.Client()
c.retrieve(    
	'reanalysis-era5-single-levels',    
	{
		'product_type': 'reanalysis',        
		'variable': ['surface_latent_heat_flux', 'surface_sensible_heat_flux',],        
		'year': ['2018'],        
		'month': ['01'],        
		'day': ['31'],        
		'time': ['19:00','20:00','21:00','22:00','23:00'],        
		'format': 'netcdf',    
},'ERA5_slhf_sshf_01.nc')
    
c.retrieve(   
	'reanalysis-era5-single-levels',    
	{
		'product_type': 'reanalysis',        
		'variable': ['surface_latent_heat_flux', 'surface_sensible_heat_flux',],        
		'year': ['2018'],        
		'month': ['02'],        
		'day': ['01'],        
		'time': ['00:00', '01:00', '02:00', '03:00', '04:00', '05:00',],        
		'format': 'netcdf',    
	},'ERA5_slhf_sshf_02.nc')

Set record dimension in each file (this is required, as it the dimension to merge the individual files along):

ncks --mk_rec_dmn time ERA5_slhf_sshf_01.nc ERA5_slhf_sshf_01_time_rec.nc 
ncks --mk_rec_dmn time ERA5_slhf_sshf_02.nc ERA5_slhf_sshf_02_time_rec.nc

Note that the data are still packed - they need to be unpacked before they can be merged:

ncpdq -U ERA5_slhf_sshf_01_time_rec.nc ERA5_slhf_sshf_01_time_rec_unpack.nc 
ncpdq -U ERA5_slhf_sshf_02_time_rec.nc ERA5_slhf_sshf_02_time_rec_unpack.nc

Then they can be successfully merged:

ncrcat ERA5_slhf_sshf_01_time_rec_unpack.nc ERA5_slhf_sshf_02_time_rec_unpack.nc  ERA5_slhf_sshf_0102_time_rec_unpack.nc

This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS) and the Copernicus Climate Change Service (C3S).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS and C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.

The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view.

Space shortcuts

Page tree

For GRIB:

For NetCDF (unsupported by ECMWF)

This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS) and the Copernicus Climate Change Service (C3S).

The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view.

Related articles