In this test case we add the WIGOS station information to a synop message. The program is tested with two SYNOP messages.


This test case differs from the Brazilian one as the input and the output files are already BUFR files but

with different BUFR sequences ( the output file must have the WIGOS sequence 301150).


As a test to assess the problem, we have created the following code

#!/usr/bin/env python
from eccodes import *
import argparse 
import pandas as pd 


'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction
 
This is a test program to encode Wigos Synop
requires
 
1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7
 
To run the program
 
   ./wigosMultisubset.py  -i synop_multi_subset.bufr -o out_synop_multisubset.bufr  -w WIGOS_IDENTIFIERS.csv 
    
Uses BUFR version 4 template  and adds the WIGOS Identifier 301150
REQUIRES TablesVersionNumber above 28
 
Author : Roberto Ribas Garcia ECMWF 09/09/2019
'''


def read_cmdline():
    '''
    reads the command line to get the input ascii filename and the output bufr file
        usage
            prog  -i <input_bufr_file> -o <output_bufr_file> -w <wigos_csv_info>
    '''
    p = argparse.ArgumentParser()
    p.add_argument('-i', '--input', help = 'input BUFR filename')
    p.add_argument('-o', '--output', help = 'output BUFR filename')
    p.add_argument("-w", "--wigoscodes",help="csv with the station codes")
    args = p.parse_args()
    return args


def read_wigosInfo(wigosCSVFile):
    '''
    dtype={"wigosLocalIdentifierCharacter":object} forces the column to be string compatible
    with the wigosLocalIdentifierCharacter key in bufr. This column in the  CSV must be created as a TEXT
    '''
    df=pd.read_csv(wigosCSVFile,sep=",",dtype={"wigosLocalIdentifierCharacter":object})
    return df 

def main():
    '''
    reads the arguments from the command line 
     -i  input bufr file
     -o  output bufr file
    -w   wigos information ( csv containing the station Name and wigos information ( wigosLocalIdentifierCharacter etc)
    '''
    args=read_cmdline()
    inputFileName=args.input 
    outputFilename=args.output 
    wigosFile=args.wigoscodes 
    '''
    reads the wigos information into a pandas dataframe that is queried 
    for each station to retrieve the station's wigos information 
    '''
    dfWigosInfo=read_wigosInfo(wigosFile)
    
    fin=open(inputFileName,"rb")
    ibid=codes_bufr_new_from_file(fin)
    
    codes_set(ibid,"unpack",1)
    inUE=codes_get_array(ibid,"unexpandedDescriptors")
    nsubsets=codes_get(ibid,"numberOfSubsets")
    
    masterTablesVN=codes_get(ibid,"masterTablesVersionNumber")
    # change the masterTablesVersionNumber if is below 29 ( otherwise the WIGOS sequence is not present)
    if masterTablesVN<28:
        masterTablesVN=28
    outUE=inUE.tolist()
    # update the unexpandedDescriptors ( BUFR sequence) to add the WIGOS data 
    outUE.insert(0,301150)
    fout=open(outputFilename,"wb")
    obid=codes_bufr_new_from_samples("BUFR4")
    ### important, use master tables version number above 28 as they contain WIGOS keys
    # otherwise it won't work
    codes_set(obid, 'masterTablesVersionNumber', masterTablesVN)
    # set the unexpandedDescriptors of the output file with the new sequence 301150 (WIGOS) + synop sequence 
    #from Input message
    # IMPORTANT, read the number of subsets 
    codes_set(obid,"numberOfSubsets",nsubsets)
    codes_set_array(obid,"unexpandedDescriptors",outUE)
    # here wigos information is added, the stationName is used
    # to query the dfWigosInfo dataframe and retrieve the station Wigos information (wigosLocalIdentifierCharacter etc)
    for i in range(0,nsubsets):
        stationKey="#{0}#stationOrSiteName".format(i+1)
        stationName=codes_get(ibid,stationKey)
        dfo=dfWigosInfo.query("station=='{0}'".format(stationName))
        key="#{0}#wigosIdentifierSeries".format(i+1)
        codes_set(obid, key,int(dfo["wigosIdentifierSeries"].values[0]) )
        key="#{0}#wigosIssuerOfIdentifier".format(i+1)
        codes_set(obid, key, int(dfo["wigosIssuerOfIdentifier"].values[0]))
        key="#{0}#wigosIssueNumber".format(i+1)
        value=dfo["wigosIssueNumber"].values[0]
        codes_set(obid, key, value)
        key="#{0}#wigosLocalIdentifierCharacter".format(i+1)
        value=dfo["wigosLocalIdentifierCharacter"].values[0]
        codes_set(obid,key,str(value))
    
   
    # copies the data from the input message ( ibid) to the output message obid
    codes_bufr_copy_data(ibid,obid)
    # write to output file ( packing is not needed here as copy_data does it implicitly.)
    codes_write(obid,fout)
    # release the obid and ibid bufr handles 
    codes_release(obid)
    codes_release(ibid)
    fout.close()
    fin.close()
    
    
    
if __name__=="__main__":
    main()

To run this program ecCodes ( above version 2.8) is required. This program was run with ecCodes version 2.12.5.

A test WIGOS_IDENTIFIERS.csv is created to test that the WIGOS keys are properly populated.

./wigosMultisubset.py -i synop_multi_subset.bufr -o aa.b -w WIGOS_IDENTIFIERS.csv


The program workflow is the following

1) read the input BUFR message and retrieve different keys

         unexpandedDescriptors  that contains the input BUFR sequence ( synop). The list of unexpandedDescriptors ( BUFR sequence) is updated to add the WIGOS sequence 301150 in front of the list

        numberOfSubsets    needed after to allocate space for the data in the output message

        masterTablesVersionNumber  this key is needed, if the masterTablesVersionNumber is below 28 ( does not contain the WIGOS sequence) then is set to 28 to make sure ecCodes finds the WIGOS sequence.

2) Once the unexpandedDescriptors sequence is updated with the WIGOS sequence, we open an output file, create a BUFR handle from a BUFR4 sample and set the information ( masterTablesVersionNumber, numberOfSubets) and  ( WIGOS information that comes  from the WIGOS_IDENTIFIERS.csv see below Important Notes  ).

This WIGOS information is read at the beginning from a CSV file (WIGOS_IDENTIFIERS.csv) and stored in a pandas dataframe dfWigosInfo, then for each station read from the input file, we query the dfWigosInfo for that particular

station and retrieve the different WIGOS fields needed for the output BUFR.


The rest of the information is copied from the input BUFR to the output BUFR by using codes_bufr_copy_data(ibid,obid) that copies the common keys from the ibid to obid.

3) once the common keys are copied and the new WIGOS keys populated properly, we just write the obid handle to the output file and release the handles ibid,obid to avoid exhausting the system's memory.

Important notes

1) The masterTablesVersionNumber must be above 28 otherwise no WIGOS sequence is available.

2) The file  WIGOS_IDENTIFIERS.csv contains individual keys for the WIGOS , as mentioned before the WigosLocalIdentifierCharacter has to be encoded as TEXT  when creating the CSV file, any other

keys that contain 0 leading values, should be encoded as TEXT as well to prevent the non leading 0 being removed as they would be treated as numeric fields.


The files used are attached here, the  out_mutisubsets.bufr contains the resulting output ( with WIGOS identifiers) for the synop_multi_subset.bufr.

The file out_singlesubset.bufr contains the  resulting output for the synop.bufr file.



test files and WIGOS csv information.


WIGOS_IDENTIFIERS.csvsynop.bufrsynop_multi_subset.bufr

Output files

out_synop.bufr

out_synop_multisubset.bufr



The output of out_synop.bufr  contains the WIGOS information

      
          "key" : "wigosIdentifierSeries",
          "value" : 0,
          "units" : "Numeric"
        },
        [

          {
            "key" : "wigosIssuerOfIdentifier",
            "value" : 20000,
            "units" : "Numeric"
          },
          [

            {
              "key" : "wigosIssueNumber",
              "value" : 0,
              "units" : "Numeric"
            },
            [

              {
                "key" : "wigosLocalIdentifierCharacter",
                "value" : "10015",
                "units" : "CCITT IA5"
              },
              [

                {
                  "key" : "blockNumber",
                  "value" : 10,
                  "units" : "Numeric"

And the output of the out_synop_multisubset.bufr contains the WIGOS keys for each of the subsets

  
 {
          "key" : "wigosIdentifierSeries",
          "value" : 0,
          "units" : "Numeric"
        },
        [

          {
            "key" : "wigosIssuerOfIdentifier",
            "value" : 20000,
            "units" : "Numeric"
          },
          [

            {
              "key" : "wigosIssueNumber",
              "value" : 0,
              "units" : "Numeric"
            },
            [

              {
                "key" : "wigosLocalIdentifierCharacter",
                "value" : "01027",
                "units" : "CCITT IA5"
              },
              [



DISCLAIMER: This software is intended for testing purposes only.  Feedback would be appreciated but technical support can only be given if our workload permits.







Write a comment…