...

It describes the blacklist language as well as its usage in IFS

Contents

• 1 Introduction User's Guide to Blacklisting
• 2 User's Guide to Blacklisting2 The Blacklist language
  • 2.1 Variables
    • 2.1.1 Report characteristics
    • 2.1.2 Model/first guess characteristics
    • 2.1.3 Observation characteristics
  • 2.2 Keywords
  • 2.3 Statements and operators
    • 2.3.1 IF-statement syntax
    • 2.3.2 List of the simple operators
    • 2.3.3 List of more complex operators
  • 2.4 Built-in functions
  • 2.5 Actions
  • 2.6 Variable declaration
• 3 Operational and experimental use of blacklist
  • 3.1 Location of blacklist files
  • 3.2 Some guidelines
• 4 Creating new blacklist file
  • 4.1 Usage of the blcomp
  • 4.2 Conversion from old to new blacklist
  • 4.3 C-code generation
  • 4.4 Linking with an application
  • 4.5 Combining conversion and object generation
  • 4.6 User interface
• 5 Examples
  • 5.1 A simple example
  • 5.2 A more complex example
  • 5.3 Adding completely new variable to the system

1 Introduction

In the operational suite on Cray computer, the blacklist was basically a list of undesired stations to be excluded from the analysis in operations, and usually in prepan experiments, too, based on monthly monitoring by the Operations Department. The technique for blacklisting has been streamlined as a part of the migration of operational codes from Cray to Fujitsu.

...

This guide comprehensively describes the format of the blacklist language developed at ECMWF during the migration project in 1995-96 based on initial idea by Mats Hamrud.

2 The Blacklist language

The way the blacklisting now works in the IFS context is as follows. One edits a blacklist file which is written in a specific format. That file is then converted into a subroutine (C language) using the blacklist compiler. The subroutine is then compiled and linked into the executable. This external routine is called from the IFS with a list of arguments in the observation screening run. IFS then receives a few flags telling whether to reject or accept this station or variable for assimilation. The following example will clarify the consepts used in blacklisting.

...

Variables get their values from IFS. These are compared against the keywords or values given in the blacklist. If the blacklist rule is true, fail-function takes action activating blacklisting flags and returning back to the calling routine in IFS. Note that the blacklist language is case insensitive and no column orientation is required.

2.1 Variables

A list of variables that are currently defined in IFS is given below. Adding new variables, see for 5.3.
 

2.1.1 Report characteristics

The up to date list of variables related to observation header and model fields can be found on HPCF in the external file of our blacklist (for instance /home/rd/rdx/data/37r3/an/external_bl_mon_monit.b for CY37R3).

2.1.2 Model/first guess characteristics

2.1.3 Observation characteristics

External variables (SPECIAL, i.e. related to obs. body entry only)

2.2 Keywords

Keywords are fixed values against which certain variables are compared. They should be consistent with the IFS definitions. A list of keywords that are currently defined in the blacklist (in the external file of our blacklist). Adding new keywords is straightforward.

2.3 Statements and operators

2.3.1 IF-statement syntax

The IF-statement syntax (note the semicolon (;) after each statement):

...

2.3.2 List of the simple operators

A list of operators that are currently defined in the Blacklist-language:

2.3.3 List of more complex operators

Somewhat more complex operators can also be used to simplify coding. For example the compound AND-operators below:

2.4 Built-in functions

The Blacklist-language also contains some built-in functions. They are listed below:

...

where the refdeg is radius of the circle on the Earth with the (reflat, reflon) as a center point of the circle. The (LAT, LON) is the position of the observation to be checked, i.e. LAT and LON of the report. All values are given in degrees. See also picture 2.1.

    
Figure 2.1: Schematic view of the rad()-function parameters.

...

1. Convert all degrees to radians
2. Calculate angle distance (in radians) relative to the center point

   obsdeg = acos( cos(reflat) cos(LAT) cos(LON-reflon) + 
                  sin(reflat) sin(LAT) )

3. Return one from rad, if , otherwise zero.

2.5 Actions

Finally, perhaps the most important function fail(). It returns information back to the application.

...

There is a range of values for ZCMCCC, and together with other information in the quality control, and a value less than one may still lead to the use of this variable in the assimilation. The inclusion of this option of non-strict blacklisting increases flexibility of the use of observations.

2.6 Variable declaration

Variable declaration has to be performed, if data will be passed from an application (like IFS) into the blacklist. This is normally done through external-declaration (see for 4.2 or 5.1). Also, selected variables can be protected by defining them as constants.

...

The simplest variable declaration is an assignment operation.

3 Operational and experimental use of blacklist

3.1 Location of blacklist files

3.2 Some guidelines

Please do not place any station identifiers into the data selection part of the blacklist. Instead, have them in the monthlt monitoring part. By this way we can have as few changes as possible in the data selection part and make e.g. re-analysis much easier.

After any modifications to the blacklist, please remember to recompile (preferably on a workstation) to check for syntax errors.

4 Creating new blacklist file

Blacklist compilation is fully controlled by the script called blcomp. It has the following capabilities:

...

• Optionally convert from an old ASCII blacklist format to a new format

   

• Check the syntax of a given blacklist

   

• Create C-language file ( C_code.c) catered for observation processing

   

• C-compile the C-file to create linkable object

4.1 Usage of the blcomp

The blcomp-script has the following usage:

...

By giving blcomp without arguments you will get the usage. If you fail to do this, check for your setting of the PATH-environment variable.

4.2 Conversion from old to new blacklist

Conversion from old to new and syntax checking of the new BLACKLIST-file can be accomplish in the following way:

...

This is exactly how the data selection part comes in in the production run, where instead of my_own_file is data selection part.

4.3 C-code generation

Enabling fast blacklist handling the blacklist file is always converted into an object file ( .o) meant to be linked with the (Fortran-)application (like IFS) in conjunction with the blacklist object library (normally libbl95.a).

...

    
blcomp -c blacklist_file.b
  or
blcomp -c blacklist_file.B

4.4 Linking with an application

A Fortran-application (IFS) interfaces the blacklist via two subroutines:

...

The exact location of the blacklist library can be found via command:

blcomp -L

4.5 Combining conversion and object generation

If no data selection part is needed, one can combine conversion from old to new blacklist and object code generation described above:

...

blcomp -c -o old_text_blacklist_file newfile.b  
  or
blcomp -c -o old_text_blacklist_file newfile.B

4.6 User interface

It is always recommended to (cold-)compile a modified blacklist on a workstation to check for syntax errors. If any errors are detected, the blcomp-command attempts to open an editor session and jump directly to the line where the (first) error occurred.

Sometimes this facility is not desirable and can be disabled by using -i flag in the blcomp-command.

5 Examples

The blacklist file is normally about 1 000 lines long. In order not to confuse readers, we will explain here with very short examples what can be done with the blacklist

5.1 A simple example

A fraction of an old blacklist ( old) looks like as follows:

...

if ( OBSTYP = synop ) then
   if   VARIAB in ( z, ps )
   and  STATID = "    3ELC"
   then fail(); endif;

   if   VARIAB in ( z, ps, u10m, v10m )
   and  STATID = "   ELBX3"
   then fail(); endif;

return; endif;

if ( OBSTYP = airep ) then
   if  (VARIAB = t)
   and  STATID in ( "  N503US", "  UAL...")
   then fail(); endif;

return; endif;

if ( OBSTYP = satob ) then
   if   STATID in ( "     0//", "     024")
   then fail(); endif;

return; endif;

if ( OBSTYP = dribu ) then
   if   VARIAB in ( z, ps, u, v )
   and  STATID = "   46527"
   then fail(); endif;

return; endif;

if ( OBSTYP = temp ) then
   if  (VARIAB = z)
   and  STATID = "    ERES"
   then fail(); endif;

return; endif;

if ( OBSTYP = pilot ) then
   if   VARIAB in ( u, v )
   and  STATID = "   08221"
   then fail(); endif;

return; endif;

if ( OBSTYP = satem ) then
   if       STATID = "     201"
   then fail(); endif;

return; endif;

5.2 A more complex example

The Blacklist compiler will generate quite a compact and readable code from the following excerpt:

...

if ( OBSTYP = synop ) then
   if   VARIAB in ( z, ps )
   and  STATID in ( "    ATQM", "    ATRK", "    ATSR", "    C6BB", "    C6QK")
   then fail(); endif;

return; endif;

if ( OBSTYP = airep ) then
   if      ( 50 >= PRESS >= 10 )
   and  STATID = "   AN..."
   then fail(); endif;

   if     (  ( LAT < -90 or LAT > 90 )   or ( -80 < LON < -40 ) )
   and  STATID = "   NWA74"
   then fail(); endif;

return; endif;

if ( OBSTYP = satob ) then
   if     (  ( LAT < -50 or LAT > 50 )   or ( -170 < LON < 90 ) )
   and  STATID = "     104"
   then fail(); endif;

   if     (  ( LAT < -50 or LAT > 50 )   or ( LON < -50 or LON > 50 ) )
   and ( 1000 >= PRESS >= 401 )
   and  STATID = "     035"
   then fail(); endif;

return; endif;

if ( OBSTYP = temp ) then
   if  (VARIAB = z)
   and ( 100 >= PRESS >= 10 )
   and ( 110000 <= TIME <= 130000 )
   and  STATID = "   20674"
   then fail(); endif;

   if   VARIAB in ( u, v )
   and ( 50000 <= TIME <= 70000 )
   and  STATID = "   40179"
   then fail(); endif;

return; endif;

if ( OBSTYP = pilot ) then
   if   VARIAB in ( u, v )
   and ( 50000 <= TIME <= 70000 )
   and  STATID = "   40179"
   then fail(); endif;

return; endif;

5.3 Adding completely new variable to the system

The current definition of variables can be checked from IFS source code in obs_preproc/blinit.F90. Adding new variables requires:

...

It describes the blacklist language as well as its usage in IFS 

Contents

• 1 Introduction
• 2 The Blacklist language
  • 2.1 Variables
    • 2.1.1 Report characteristics
    • 2.1.2 Model/first guess characteristics
    • 2.1.3 Observation characteristics
  • 2.2 Keywords
  • 2.3 Statements and operators
    • 2.3.1 IF-statement syntax
    • 2.3.2 List of the simple operators
    • 2.3.3 List of more complex operators
  • 2.4 Built-in functions
  • 2.5 Actions
  • 2.6 Variable declaration
• 3 Operational and experimental use of blacklist
  • 3.1 Location of blacklist files
  • 3.2 Some guidelines
• 4 Creating new blacklist file
  • 4.1 Usage of the blcomp
  • 4.2 Conversion from old to new blacklist
  • 4.3 C-code generation
  • 4.4 Linking with an application
  • 4.5 Combining conversion and object generation
  • 4.6 User interface
• 5 Examples
  • 5.1 A simple example
  • 5.2 A more complex example
  • 5.3 Adding completely new variable to the system

1 Introduction

In the operational suite on Cray computer, the blacklist was basically a list of undesired stations to be excluded from the analysis in operations, and usually in prepan experiments, too, based on monthly monitoring by the Operations Department. The technique for blacklisting has been streamlined as a part of the migration of operational codes from Cray to Fujitsu.

...

This guide comprehensively describes the format of the blacklist language developed at ECMWF during the migration project in 1995-96 based on initial idea by Mats Hamrud.

2 The Blacklist language

The way the blacklisting now works in the IFS context is as follows. One edits a blacklist file which is written in a specific format. That file is then converted into a subroutine (C language) using the blacklist compiler. The subroutine is then compiled and linked into the executable. This external routine is called from the IFS with a list of arguments in the observation screening run. IFS then receives a few flags telling whether to reject or accept this station or variable for assimilation. The following example will clarify the consepts used in blacklisting.

...

Variables get their values from IFS. These are compared against the keywords or values given in the blacklist. If the blacklist rule is true, fail-function takes action activating blacklisting flags and returning back to the calling routine in IFS. Note that the blacklist language is case insensitive and no column orientation is required.

2.1 Variables

A list of variables that are currently defined in IFS is given below. Adding new variables, see for 5.3.
 

2.1.1 Report characteristics

The up to date list of variables related to observation header and model fields can be found on HPCF in the external file of our blacklist (for instance /home/rd/rdx/data/37r3/an/external_bl_mon_monit.b for CY37R3).

2.1.2 Model/first guess characteristics

2.1.3 Observation characteristics

External variables (SPECIAL, i.e. related to obs. body entry only)

2.2 Keywords

Keywords are fixed values against which certain variables are compared. They should be consistent with the IFS definitions. A list of keywords that are currently defined in the blacklist (in the external file of our blacklist). Adding new keywords is straightforward.

2.3 Statements and operators

2.3.1 IF-statement syntax

The IF-statement syntax (note the semicolon (;) after each statement):

...

2.3.2 List of the simple operators

A list of operators that are currently defined in the Blacklist-language:

2.3.3 List of more complex operators

Somewhat more complex operators can also be used to simplify coding. For example the compound AND-operators below:

2.4 Built-in functions

The Blacklist-language also contains some built-in functions. They are listed below:

...

where the refdeg is radius of the circle on the Earth with the (reflat, reflon) as a center point of the circle. The (LAT, LON) is the position of the observation to be checked, i.e. LAT and LON of the report. All values are given in degrees. See also picture 2.1.

    
Figure 2.1: Schematic view of the rad()-function parameters.

...

1. Convert all degrees to radians
2. Calculate angle distance (in radians) relative to the center point

   obsdeg = acos( cos(reflat) cos(LAT) cos(LON-reflon) + 
                  sin(reflat) sin(LAT) )

3. Return one from rad, if , otherwise zero.

2.5 Actions

Finally, perhaps the most important function fail(). It returns information back to the application.

...

There is a range of values for ZCMCCC, and together with other information in the quality control, and a value less than one may still lead to the use of this variable in the assimilation. The inclusion of this option of non-strict blacklisting increases flexibility of the use of observations.

2.6 Variable declaration

Variable declaration has to be performed, if data will be passed from an application (like IFS) into the blacklist. This is normally done through external-declaration (see for 4.2 or 5.1). Also, selected variables can be protected by defining them as constants.

...

The simplest variable declaration is an assignment operation.

3 Operational and experimental use of blacklist

3.1 Location of blacklist files

3.2 Some guidelines

Please do not place any station identifiers into the data selection part of the blacklist. Instead, have them in the monthlt monitoring part. By this way we can have as few changes as possible in the data selection part and make e.g. re-analysis much easier.

After any modifications to the blacklist, please remember to recompile (preferably on a workstation) to check for syntax errors.

4 Creating new blacklist file

Blacklist compilation is fully controlled by the script called blcomp. It has the following capabilities:

...

• Optionally convert from an old ASCII blacklist format to a new format

   

• Check the syntax of a given blacklist

   

• Create C-language file ( C_code.c) catered for observation processing

   

• C-compile the C-file to create linkable object

4.1 Usage of the blcomp

The blcomp-script has the following usage:

...

By giving blcomp without arguments you will get the usage. If you fail to do this, check for your setting of the PATH-environment variable.

4.2 Conversion from old to new blacklist

Conversion from old to new and syntax checking of the new BLACKLIST-file can be accomplish in the following way:

...

This is exactly how the data selection part comes in in the production run, where instead of my_own_file is data selection part.

4.3 C-code generation

Enabling fast blacklist handling the blacklist file is always converted into an object file ( .o) meant to be linked with the (Fortran-)application (like IFS) in conjunction with the blacklist object library (normally libbl95.a).

...

    
blcomp -c blacklist_file.b
  or
blcomp -c blacklist_file.B

4.4 Linking with an application

A Fortran-application (IFS) interfaces the blacklist via two subroutines:

...

The exact location of the blacklist library can be found via command:

blcomp -L

4.5 Combining conversion and object generation

If no data selection part is needed, one can combine conversion from old to new blacklist and object code generation described above:

...

blcomp -c -o old_text_blacklist_file newfile.b  
  or
blcomp -c -o old_text_blacklist_file newfile.B

4.6 User interface

It is always recommended to (cold-)compile a modified blacklist on a workstation to check for syntax errors. If any errors are detected, the blcomp-command attempts to open an editor session and jump directly to the line where the (first) error occurred.

Sometimes this facility is not desirable and can be disabled by using -i flag in the blcomp-command.

5 Examples

The blacklist file is normally about 1 000 lines long. In order not to confuse readers, we will explain here with very short examples what can be done with the blacklist

5.1 A simple example

A fraction of an old blacklist ( old) looks like as follows:

...

if ( OBSTYP = synop ) then
   if   VARIAB in ( z, ps )
   and  STATID = "    3ELC"
   then fail(); endif;

   if   VARIAB in ( z, ps, u10m, v10m )
   and  STATID = "   ELBX3"
   then fail(); endif;

return; endif;

if ( OBSTYP = airep ) then
   if  (VARIAB = t)
   and  STATID in ( "  N503US", "  UAL...")
   then fail(); endif;

return; endif;

if ( OBSTYP = satob ) then
   if   STATID in ( "     0//", "     024")
   then fail(); endif;

return; endif;

if ( OBSTYP = dribu ) then
   if   VARIAB in ( z, ps, u, v )
   and  STATID = "   46527"
   then fail(); endif;

return; endif;

if ( OBSTYP = temp ) then
   if  (VARIAB = z)
   and  STATID = "    ERES"
   then fail(); endif;

return; endif;

if ( OBSTYP = pilot ) then
   if   VARIAB in ( u, v )
   and  STATID = "   08221"
   then fail(); endif;

return; endif;

if ( OBSTYP = satem ) then
   if       STATID = "     201"
   then fail(); endif;

return; endif;

5.2 A more complex example

The Blacklist compiler will generate quite a compact and readable code from the following excerpt:

...

if ( OBSTYP = synop ) then
   if   VARIAB in ( z, ps )
   and  STATID in ( "    ATQM", "    ATRK", "    ATSR", "    C6BB", "    C6QK")
   then fail(); endif;

return; endif;

if ( OBSTYP = airep ) then
   if      ( 50 >= PRESS >= 10 )
   and  STATID = "   AN..."
   then fail(); endif;

   if     (  ( LAT < -90 or LAT > 90 )   or ( -80 < LON < -40 ) )
   and  STATID = "   NWA74"
   then fail(); endif;

return; endif;

if ( OBSTYP = satob ) then
   if     (  ( LAT < -50 or LAT > 50 )   or ( -170 < LON < 90 ) )
   and  STATID = "     104"
   then fail(); endif;

   if     (  ( LAT < -50 or LAT > 50 )   or ( LON < -50 or LON > 50 ) )
   and ( 1000 >= PRESS >= 401 )
   and  STATID = "     035"
   then fail(); endif;

return; endif;

if ( OBSTYP = temp ) then
   if  (VARIAB = z)
   and ( 100 >= PRESS >= 10 )
   and ( 110000 <= TIME <= 130000 )
   and  STATID = "   20674"
   then fail(); endif;

   if   VARIAB in ( u, v )
   and ( 50000 <= TIME <= 70000 )
   and  STATID = "   40179"
   then fail(); endif;

return; endif;

if ( OBSTYP = pilot ) then
   if   VARIAB in ( u, v )
   and ( 50000 <= TIME <= 70000 )
   and  STATID = "   40179"
   then fail(); endif;

return; endif;

5.3 Adding completely new variable to the system

The current definition of variables can be checked from IFS source code in obs_preproc/blinit.F90. Adding new variables requires:

...