Italian Met Service has assigned a WIGOS ID (0-380-8-1) to the new Antarctic station called “Paola”.
This means that “Paola” has no WMO traditional block number and station number.
https://oscar.wmo.int/surface/#/search/station/stationReportDetails/0-380-8-1.
This test case differs from the Brazilian one as the input and the output files are already BUFR files but
with different BUFR sequences ( the output file must have the WIGOS sequence 301150).
As a test to assess the problem, we have created the following code
#!/usr/bin/env python
from eccodes import *
import argparse
import pandas as pd
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction
This is a test program to encode Wigos Synop
requires
1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7
To run the program
./wigosItalyMultisubset.py -i synop_multi_subset.bufr -o test_multisubset.bufr -w WIGOS_IDENTIFIERS.csv
Adds BUFR version 4 template 301150 to a synop
Author : Roberto Ribas Garcia ECMWF 09 Sep 2019
'''
def read_cmdline():
'''
reads the command line to get the input ascii filename and the output bufr file
usage
prog -i <input_bufr_file> -o <output_bufr_file> -w <wigos_csv_info>
'''
p = argparse.ArgumentParser()
p.add_argument('-i', '--input', help = 'input BUFR filename')
p.add_argument('-o', '--output', help = 'output BUFR filename')
p.add_argument("-w", "--wigoscodes",help="csv with the station codes")
args = p.parse_args()
return args
def read_wigosInfo(wigosCSVFile):
'''
reads the WIGOS information from a CSV file like this
station,wigosIdentifierSeries,wigosIssuerOfIdentifier,wigosIssueNumber,wigosLocalIdentifierCharacter
TROMSO-HOLT,0,76,0,0-20000-0-01027
PASVIK,0,77,0,0-20000-0-01084
KVITHAMAR,0,78,0,0-20000-0-01270
'''
df=pd.read_csv(wigosCSVFile,sep=",")
return df
def main():
'''
reads the arguments from the command line
-i input bufr file
-o output bufr file
-w wigos information ( csv containing the station Name and wigos information ( wigosLocalIdentifierCharacter etc)
'''
args=read_cmdline()
inputFileName=args.input
outputFilename=args.output
wigosFile=args.wigoscodes
'''
reads the wigos information into a pandas dataframe that is queried
for each station to retrieve the station's wigos information
'''
dfWigosInfo=read_wigosInfo(wigosFile)
fin=open(inputFileName,"rb")
ibid=codes_bufr_new_from_file(fin)
codes_set(ibid,"unpack",1)
inUE=codes_get_array(ibid,"unexpandedDescriptors")
nsubsets=codes_get(ibid,"numberOfSubsets")
masterTablesVN=codes_get(ibid,"masterTablesVersionNumber")
# change the masterTablesVersionNumber if is below 28 ( otherwise the WIGOS sequence is not present)
if masterTablesVN<28:
masterTablesVN=28
outUE=inUE.tolist()
# update the unexpandedDescriptors ( BUFR sequence) to add the WIGOS data
outUE.insert(0,301150)
fout=open(outputFilename,"wb")
# create a new message from SAMPLE BUFR4 template
obid=codes_bufr_new_from_samples("BUFR4")
### important, use master tables version number above 28 as they contain WIGOS keys
# otherwise it won't work
codes_set(obid, 'masterTablesVersionNumber', masterTablesVN)
# set the unexpandedDescriptors of the output file with the new sequence 301150 (WIGOS) + synop sequence
#from Input message
# IMPORTANT, read the number of subsets
codes_set(obid,"numberOfSubsets",nsubsets)
codes_set_array(obid,"unexpandedDescriptors",outUE)
# here wigos information is added, the stationName is used
# to query the dfWigosInfo dataframe and retrieve the station Wigos information (wigosLocalIdentifierCharacter etc)
for i in range(0,nsubsets):
stationKey="#{0}#stationOrSiteName".format(i+1)
stationName=codes_get(ibid,stationKey)
dfo=dfWigosInfo.query("station=='{0}'".format(stationName))
key="#{0}#wigosIdentifierSeries".format(i+1)
codes_set(obid, key,int(dfo["wigosIdentifierSeries"].values[0]) )
key="#{0}#wigosIssuerOfIdentifier".format(i+1)
codes_set(obid, key, int(dfo["wigosIssuerOfIdentifier"].values[0]))
key="#{0}#wigosIssueNumber".format(i+1)
value=dfo["wigosIssueNumber"].values[0]
codes_set(obid, key, value)
key="#{0}#wigosLocalIdentifierCharacter".format(i+1)
value=dfo["wigosLocalIdentifierCharacter"].values[0]
codes_set(obid,key,value)
# copies the data from the input message ( ibid) to the output message obid
codes_bufr_copy_data(ibid,obid)
# pack and write to output file
codes_set(obid,"pack",1)
codes_write(obid,fout)
# release the obid and ibid bufr handles
codes_release(obid)
codes_release(ibid)
fout.close()
fin.close()
if __name__=="__main__":
main()
To run this program ecCodes ( above version 2.8) is required. This program was run with eccodes version 2.12.5.
A test WIGOS_IDENTIFIERS.csv is created to test that the WIGOS keys are properly populated.
./wigosItalyMultisubset.py -i synop_multi_subset.bufr -o aa.b -w WIGOS_IDENTIFIERS.csv
The program workflow is the following
1) read the input BUFR message and retrieve different keys
unexpandedDescriptors that contains the input BUFR sequence ( synop). The list of unexpandedDescriptors ( BUFR sequence) is updated to add the WIGOS sequence 301150 in front of the list
numberOfSubsets needed after to allocate the data
masterTablesVersionNumber this key is needed, if the masterTablesVersionNumber is below 28 ( does not contain the WIGOS sequence) then is set to 28 to make sure ecCodes finds the WIGOS sequence.
2) Once the unexpandedDescriptors sequence is updated with the WIGOS sequence, we open an output file, create a BUFR handle from a BUFR4 sample and set the information ( masterTablesVersionNumber, numberOfSubets) and ( WIGOS information that comes from the WIGOS_IDENTIFIERS.csv ).
This WIGOS information is read at the beginning from a CSV file (WIGOS_IDENTIFIERS.csv) and stored in a pandas dataframe dfWigosInfo, then for each station read from the input file, we query the dfWigosInfo for that particular
station and retrieve the different WIGOS fields needed for the output BUFR.
The rest of the information is copied from the input BUFR to the output BUFR by using codes_bufr_copy_data(ibid,obid) that copies the common keys from the ibid to obid.
3) once the common keys are copied and the new WIGOS keys populated properly, we just write the obid handle to the output file and release the handles ibid,obid to avoid exhausting the system's memory.
The files used are attached here, the test_mutisubsets.bufr contains the resulting output ( with WIGOS identifiers) for the synop_multi_subset.bufr.
The file test_singlesubset.bufr contains the resulting output for the synop.bufr file
The output of test_singlesubset.bufr contains the WIGOS information
{
"key" : "subsetNumber",
"value" : 1
},
{
"key" : "wigosIdentifierSeries",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosIssuerOfIdentifier",
"value" : 88,
"units" : "Numeric"
},
[
{
"key" : "wigosIssueNumber",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosLocalIdentifierCharacter",
"value" : "0-20000-0-10015",
"units" : "CCITT IA5"
},
And the output of the test_multisubset.bufr contains the WIGOS keys for each of the subsets
},
[
{
"key" : "wigosIssueNumber",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosLocalIdentifierCharacter",
"value" : "0-20000-0-01027",
"units" : "CCITT IA5"
},
[
{
"key" : "blockNumber",
"value" : 1,
"units" : "Numeric"
},
[
{
"key" : "stationNumber",
"value" : 27,
"units" : "Numeric"
},
[
{
"key" : "stationOrSiteName",
"value" : "TROMSO-HOLT",
"units" : "CCITT IA5"