Here we show an example of encoding meteorological observations from an ASCII file. We will encode SYNOP data provided in CVS format utilising BUFR utilising BUFR template 307092 which is for sub-hourly observations.
1. Choosing the template
In order to encode meteorological observations in BUFR format, first thing to do is to decide which BUFR template we should use. This can be done by investigating WMO BUFR Table D which provides information on list of common sequences. For example, let’s assume we want to BUFR encode sub-hourly data from an automatic weather station. As you can see from the below given screenshot of Table D, the suitable template for this is 3 07 092 which is BUFR template for surface observations from n-minute period.
Expand | ||
---|---|---|
| ||
2. Investigating the template
Next thing to do is to investigate the template to have a better understanding what parameters we can encode with the template. For this, we need to refer to WMO tables (which can be accessed here) which are
- Table B Classification of elements
- Table C Data description operators
- Table D List of common sequences
Table D defines sequence descriptors which are alias for a sequence of other descriptors. For example, one of the elements within the sequence 3 07 092 is Pressure having table reference value 0 07 004. As you can see from the Table B screenshot below, Table B provides detailed information on this element such as unit, scale, reference value and data width.
...
Code Block | ||||
---|---|---|---|---|
| ||||
#!/usr/bin/env pythonpython3 # (C) Copyright 1996- ECMWF. # # This software is licensed under the terms of the Apache Licence Version 2.0 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. # # In applying this licence, ECMWF does not waive the privileges and immunities # granted to it by virtue of its status as an intergovernmental organisation # nor does it submit to any jurisdiction. from eccodes import * ibufr = codes_bufr_new_from_samples('BUFR4') # Creates a new valid message id from a BUFR sample codes_set(ibufr, 'edition', 4) # BUFR edition number codes_set(ibufr, 'masterTableNumber', 0) # BUFR master table. Zero: standard WMO FM 94 BUFR tables codes_set(ibufr, 'masterTablesVersionNumber', 31) # Version number of master table used ivalues = (307092) # Template to be used codes_set(ibufr, 'unexpandedDescriptors', ivalues) # Key name to encode the sequence number is unexpandedDescriptors fout = open('TM307092.bufr', 'wwb') # Open output file codes_write(ibufr, fout) # Write the message to output file codes_release(ibufr) # Release the BUFR message from memory fout.close() # Close the file |
...
Expand | ||
---|---|---|
| ||
|
3. Preparing the CSV file
TM307092.plain tells us what parameters (keys) we can encode with the template 307092. Now we will create a file containing our data which will be in CSV format. In this step, we need to consider both the content of 307092.plain and associated WMO tables to provide key-value pairs appropriately (units, scale, etc.). Please note that although CSV file is chosen as an example here, you may use any format to provide your data as input to your python program for BUFR encoding as long as you are able to make the program read your data.
...
The SYNOP data provided above is created just to serve as an example for this tutorial. As you may realise, it provides values for a limited number of parameters. While you are creating your input files, you should include as much information as possible from the measurements of your station and change the below given csv2bufr.py accordingly to read from the correct column and write to the correct key.
4. Encoding the BUFR
...
data
Code Block | ||||
---|---|---|---|---|
| ||||
#!/usr/bin/env pythonpython3 # (C) Copyright 1996- ECMWF. # # This software is licensed under the terms of the Apache Licence Version 2.0 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. # # In applying this licence, ECMWF does not waive the privileges and immunities # granted to it by virtue of its status as an intergovernmental organisation # nor does it submit to any jurisdiction. from eccodes import * import csv, argparse # read command line to get input filename def read_cmdline(): p = argparse.ArgumentParser() p.add_argument("--i",help=" input Ascii filename") args = p.parse_args() return args # read data from CSV file into a list def csv_read(filename): data = [] try: with open(filename) as csvfile: reader = csv.reader(csvfile, delimiter='|') for row in reader: data.append(row) except IOError as error: print(error) sys.exit(1) else: csvfile.close() return data[1:] # Encode the data from CSV into BUFR def message_encoding(FileName, fout): # reads the CSV file into a python list dataIn = csv_read(FileName) # loops over the rows of the csv file (one BUFR message for each row) for row in dataIn: bid = codes_bufr_new_from_samples('BUFR4') for ele in range(len(row)): row[ele] = row[ele].strip() try: bufr_encode(bid, row) codes_write(bid, fout) except CodesInternalError as ec: print (ec) codes_release(bid) def bufr_encode(ibufr, row): # set header keys and values codes_set(ibufr, 'edition', 4) codes_set(ibufr, 'masterTableNumber', 0) codes_set(ibufr, 'bufrHeaderCentre', 98) # 98: centre is ecmf codes_set(ibufr, 'bufrHeaderSubCentre', 0) codes_set(ibufr, 'updateSequenceNumber', 0) codes_set(ibufr, 'dataCategory', 0) # 0: Surface data - land codes_set(ibufr, 'internationalDataSubCategory', 7) # 7: n-min obs from AWS stations codes_set(ibufr, 'dataSubCategory', 7) codes_set(ibufr, 'masterTablesVersionNumber', 31) codes_set(ibufr, 'localTablesVersionNumber', 0) codes_set(ibufr, 'observedData', 1) codes_set(ibufr, 'compressedData', 0) codes_set(ibufr, 'typicalYear', int(row[0])) codes_set(ibufr, 'typicalMonth', int(row[1])) codes_set(ibufr, 'typicalDay', int(row[2])) codes_set(ibufr, 'typicalHour', int(row[3])) codes_set(ibufr, 'typicalMinute', int(row[4])) codes_set(ibufr, 'typicalSecond', 0) ivalues=(307092) codes_set(ibufr, 'unexpandedDescriptors', ivalues) # set data keys and values codes_set(ibufr, 'year', int(row[0])) codes_set(ibufr, 'month', int(row[1])) codes_set(ibufr, 'day', int(row[2])) codes_set(ibufr, 'hour', int(row[3])) codes_set(ibufr, 'minute', int(row[4])) codes_set(ibufr, 'blockNumber', int(row[5])) codes_set(ibufr, 'stationNumber', int(row[6])) codes_set(ibufr, 'longStationName',row[7].strip()) codes_set(ibufr, 'latitude', float(row[8])) codes_set(ibufr, 'longitude', float(row[9])) codes_set(ibufr, 'heightOfStationGroundAboveMeanSeaLevel', float(row[10])) codes_set(ibufr, 'pressure', float(row[11])) codes_set(ibufr, 'pressureReducedToMeanSeaLevel', float(row[12])) codes_set(ibufr, 'airTemperature', float(row[13])) codes_set(ibufr, '#1#relativeHumidity', float(row[14])) codes_set(ibufr, '#2#timePeriod', -10) # -10: Period of precipitation observation is 10 minutes codes_set(ibufr, 'totalPrecipitationOrTotalWaterEquivalent', float(row[15])) codes_set(ibufr, '#1#timeSignificance', 2) # 2: Time averaged codes_set(ibufr, '#3#timePeriod', -10) # -10: Period of wind observations is 10 minutes codes_set(ibufr, 'windDirection', float(row[16])) codes_set(ibufr, 'windSpeed', float(row[17])) codes_set(ibufr, 'pack', 1) # Required to encode the keys back in the data section def main(): cmdLine = read_cmdline() inputFilename = cmdLine.i print (inputFilename) outFilename = str(inputFilename.split('.')[0]+'.bufr') fout = open(outFilename, "wwb") message_encoding(inputFilename, fout) fout.close() print (" output file {0}".format(outFilename)) if __name__ == '__main__': main() |
...
Code Block |
---|
user@host:~> bufr_ls datain.bufr datain.bufr centre masterTablesVersionNumber localTablesVersionNumber typicalDate typicalTime masterTablesVersionNumbernumberOfSubsets ecmf localTablesVersionNumber typicalDate 31 typicalTime 0 numberOfSubsets ecmf 3120200914 140000 1 ecmf 0 31 20200914 0 140000 20200914 1 ecmf 31 0 20200914 140000 140000 1 2 of 2 messages in datain.bufr |
Related articles
Content by Label | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...