This guide comprehensively describes the format of the blacklist language developed at ECMWF during the migration project in 1995-96 based on initial idea by Mats Hamrud.

2 The Blacklist language

...

Variables get their values from IFS. These are compared against the keywords or values given in the blacklist. If the blacklist rule is true, fail-function takes action activating blacklisting flags and returning back to the calling routine in IFS. Note that the blacklist language is case insensitive and no column orientation is required.

2.1 Variables

...

A list of variables that are currently defined in IFS is given below. Adding new variables, see for 5.3.

2.1.1 Report characteristics

...

The up to date list of variables related to observation header and model fields can be found on HPCF in the external file of our blacklist (for instance /home/rd/rdx/data/37r3/an/external_bl_mon_monit.b for CY37R3).
...

2.1.2 Model/first guess characteristics

...

Variable Meaning Possible values
modps model surface pressure Real Value
modts model surface temperature Real Value
modt2m model 2 metre temperature Real Value
modtop model top level pressure (hPa) Real Value
sea_ice model sea-ice fraction Real Value
...

2.1.3 Observation characteristics

...

External variables (SPECIAL, i.e. related to obs. body entry only)
Variable Meaning Possible values
variab variable name (varno in ODB) Integer value
vert_co type of vert. coord. Integer value
press pressure (hPa) Real value
press_rl ref. level press. (hPa) Real value
ppcode synop press. code Integer value
obs_value observed value Real value
fg_departure first guess depart. Real value
obs_error observation error Real value
fg_error first guess error Real value
winchan_dep window chan dep Real value
obs_t Obs temperature at same level, for R/S only. Real value
elevation Radar elevation Real value
winchan_dep2 alternative window chan dep Real value
tausfc Surface transmittance for AIRS screening. Real value
csr_pclear percentage of clear pixel (GEOS) Real value

2.2 Keywords

...

Keywords are fixed values against which certain variables are compared. They should be consistent with the IFS definitions. A list of keywords that are currently defined in the blacklist (in the external file of our blacklist). Adding new keywords is straightforward.
...

2.3 Statements and operators

2.3.1 IF-statement syntax

The IF-statement syntax (note the semicolon (;) after each statement):

Syntax

Meaning

if (condition) then
      statement_1;
      statement_2;
      etc.
elif (condition) then
   statement_1;
   statement_2;
   etc.
else
   statement_1;
   statement_2;
   etc.
endif

IF-test with optional ELIF/ELSE-blocks.

Nested IF-tests are valid in every statement. Every IF-THEN or IF-THEN-ELSE must match an ENDIF

Condition can be any logical or arithmetic operation.

...

2.3.3 List of more complex operators

...

Somewhat more complex operators can also be used to simplify coding. For example the compound AND-operators below:

2.4 Built-in functions

The Blacklist-language also contains some built-in functions. They are listed below:

In addition, there is one special function to study whether a point is within a circular area on the Earth (e.g. to blacklist Meteosat SATOBs if they are too far away):

if (not (rad (0, 0, 45, LAT, LON))) then fail(monthly); endif;

The function is called rad() and requires five (5) arguments. It returns one (1) if the observation is within the circle, otherwise zero (0). The usage is

rad(reflat, reflon, refdeg, LAT, LON)

where the refdeg is radius of the circle on the Earth with the (reflat, reflon) as a center point of the circle. The (LAT, LON) is the position of the observation to be checked, i.e. LAT and LON of the report. All values are given in degrees. See also picture 2.1.

...

The following arithmetic is performed in the function rad():

Convert all degrees to radians

Calculate angle distance (in radians) relative to the center point

obsdeg = acos( cos(reflat) cos(LAT) cos(LON-reflon) + 
               sin(reflat) sin(LAT) )

Return one from rad, if , otherwise zero.

2.5 Actions

...

Finally, perhaps the most important function fail(). It returns information back to the application.
...
Argument#1 (arg1)
Meaning
monthly monthly monitoring (default)
constant constant blacklisting
experimental experimental blacklisting
use_emiskf_only emiskf blacklisting

...
Argument#2 (arg2)
Meaning
level
Level of seriousness of blacklisting
Range is between [0..1]. Default =1

...

When a call to the fail()-function occurs, the control is returned immediately to the calling application. Normally the application is the IFS, which will get the following (Fortran) variables updated:
Variable Type Meaning
NCMBLI Integer
Blacklisting indicator
0= not blacklisted (default)
1= monthly monitoring
2= constant monitoring
3=experimental
4= use for emiskf only
ZCMCCC Real
Seriousness of the blacklisting
0= Default if not blacklisted
1= Default if blacklisted (i.e. NCMBLI > 0)
[0.01...0.99] for non-complete blacklisting (optional)
FEEDBACK Integer
Feedback vector telling which variable(s) caused the blacklisting to occur:
0 = Blacklist line number where the fail()-function took action
1-N = Pointers to the variable indices to help to locate the responsible variables

...
There is a range of values for ZCMCCC, and together with other information in the quality control, and a value less than one may still lead to the use of this variable in the assimilation. The inclusion of this option of non-strict blacklisting increases flexibility of the use of observations.

2.6 Variable declaration

...

Variable declaration has to be performed, if data will be passed from an application (like IFS) into the blacklist. This is normally done through external-declaration (see for 4.2 or 5.1). Also, selected variables can be protected by defining them as constants.
...
The simplest variable declaration is an assignment operation.

3 Operational and experimental use of blacklist

...

3.1 Location of blacklist files

3.2 Some guidelines

...

Please do not place any station identifiers into the data selection part of the blacklist. Instead, have them in the monthlt monitoring part. By this way we can have as few changes as possible in the data selection part and make e.g. re-analysis much easier.
After any modifications to the blacklist, please remember to recompile (preferably on a workstation) to check for syntax errors.

4 Creating new blacklist file

Blacklist compilation is fully controlled by the script called blcomp. It has the following capabilities:

Optionally convert from an old ASCII blacklist format to a new format
Check the syntax of a given blacklist
Create C-language file ( C_code.c) catered for observation processing
C-compile the C-file to create linkable object

4.1 Usage of the blcomp

...

The blcomp-script has the following usage:

blcomp [-aAcCdDefiILmMnoOpSx8] blacklist_file.b (or blacklist_file.B)

...

where the flags are as follows:
...
For pure syntax checking of the new BLACKLIST-file, give:
 
blcomp blacklist_file.b
  or
blcomp blacklist_file.B
 
By giving blcomp without arguments you will get the usage. If you fail to do this, check for your setting of the PATH-environment variable. 
4.2 Conversion from old to new blacklist
...
Conversion from old to new and syntax checking of the new BLACKLIST-file can be accomplish in the following way: 
blcomp -o old_text_blacklist_file newfile.b
  or
blcomp -o old_text_blacklist_file newfile.B
...
If the suffix .B was used, then the following three (3) files are generated:
 
master file ( newfile.B)
include-file no. 1 for externals ( external_newfile.b)
include-file no. 2 for monthly part ( monthly_newfile.b)
 
The contents of the master file is simply the following two lines:
 
#include "external_newfile.b"
#include "monthly_newfile.b"
...
One way to bring in your own modifications, is to create a new master-file, for example: 
#include "external_newfile.b"
#include "my_own_file"
#include "monthly_newfile.b"
...
This is exactly how the data selection part comes in in the production run, where instead of my_own_file is data selection part. 
4.3 C-code generation
 
Enabling fast blacklist handling the blacklist file is always converted into an object file ( .o) meant to be linked with the (Fortran-)application (like IFS) in conjunction with the blacklist object library (normally libbl95.a).
Once a blacklist file (either with .b or .B suffix) is available, it can be converted to C-language file C_code.c and compiled to an object for maximum performance. This can be done as follows:
 
...
blcomp -c blacklist_file.b
  or
blcomp -c blacklist_file.B
...
 
4.4 Linking with an application
 
A Fortran-application (IFS) interfaces the blacklist via two subroutines:
 
BLACKBOX_INIT
BLACKBOX
...
The former one is responsible for initiating the variable list active by the application. And the latter one handles all burden of interfacing the blacklist file.
To link application with the blacklist software, one needs not only the C_code.o-object file, but also the blacklist library libbl95.a. Linking command is normally: 
linker application.o C_code.o /bl95path/libbl95.a other_libs
...
The exact location of the blacklist library can be found via command: 
blcomp -L
 
 
4.5 Combining conversion and object generation
 
If no data selection part is needed, one can combine conversion from old to new blacklist and object code generation described above: 
blcomp -c -o old_text_blacklist_file newfile.b  
  or
blcomp -c -o old_text_blacklist_file newfile.B
 
...
4.6 User interface
 
It is always recommended to (cold-)compile a modified blacklist on a workstation to check for syntax errors. If any errors are detected, the blcomp-command attempts to open an editor session and jump directly to the line where the (first) error occurred.
Sometimes this facility is not desirable and can be disabled by using -i flag in the blcomp-command.
 
5 Examples
...
The blacklist file is normally about 1 000 lines long. In order not to confuse readers, we will explain here with very short examples what can be done with the blacklist 
5.1 A simple example
...
A fraction of an old blacklist ( old) looks like as follows:
 
     3ELC  1 3
    ELBX3  1 333
   N503US  2 00030
   UAL...  2 00030
      024  3 33000000 033333   
      0//  3 33000000 033333
    46527  4 33300
     ERES  5          000003
    08221  6          0330
      201  7 33300000 00333
...
When compiled with blcomp -o old new.b, we get a new file new.b. The local constant variable declaration section looks as follows:
 
!
!	 Written by an automatic conversion program, version 3
!
!
!	 File converted from the file "old"
!

! FAILCODE :
const monthly = 1;
const constant = 2;
const experimental = 3;
const whitelist = 4;

! OBSTYP :
const synop = 1;
const airep = 2;
const satob = 3;
const dribu = 4;
const temp = 5;
const pilot = 6;
const satem = 7;
const paob = 8;
const scatt = 9;

! CODTYP : none

! INSTRM : none

! VARIAB :
const u = 3;
const v = 4;
const z = 1;
const dz = 57;
const rh = 29;
const q = 7;
const pwc = 9;
const rh2m = 58;
const t = 2;
const td = 59;
const t2m = 39;
const td2m = 40;
const ts = 11;
const ptend = 30;
const w = 60;
const ww = 61;
const vv = 62;
const ch = 63;
const cm = 64;
const cl = 65;
const nh = 66;
const nn = 67;
const hshs = 68;
const c = 69;
const ns = 70;
const s = 71;
const e = 72;
const tgtg = 73;
const spsp1 = 74;
const spsp2 = 75;
const rs = 76;
const eses = 77;
const is = 78;
const trtr = 79;
const rr = 80;
const jj = 81;
const vs = 82;
const ds = 83;
const hwhw = 84;
const pwpw = 85;
const dwdw = 86;
const gclg = 87;
const rhlc = 88;
const rhmc = 89;
const rhhc = 90;
const n = 91;
const snra = 92;
const ps = 110;
const dd = 111;
const ff = 112;
const rawbt = 119;
const rawra = 120;
const satcl = 121;
const scatss = 122;
const du = 5;
const dv = 6;
const u10m = 41;
const v10m = 42;
const rhlay = 19;
const auxil = 200;
const cllqw = 123;
const scatdd = 124;
const scatff = 125;

! LSMASK :
const sea = 0;
const land = 1;

! PPCODE :
const psealev = 0;
const pstalev = 1;
const g850hpa = 2;
const g700hpa = 3;
const p500gpm = 4;
const p1000gpm = 5;
const p2000gpm = 6;
const p3000gpm = 7;
const p4000gpm = 8;
const g900hpa = 9;
const g1000hpa = 10;
const g500hpa = 11;

! VERT_CO:
const pressure = 1;
const height = 2;
const tovs_cha = 3;
const scat_cha = 4;
...
The external variable definition section looks as follows:
 
! External variables (non-special):
external obstyp;
external_CHAR statid;
external codtyp;
external instrm;
external date;
external time;
external lat;
external lon;
external stalt;
external modoro;
external lsmask;
external rad;

! External variables (SPECIAL):
external variab is SPECIAL;
external vert_co is SPECIAL;
external press is SPECIAL;
external press_rl is SPECIAL;
external ppcode is SPECIAL;
external obs_value is SPECIAL;
external obs_departure is SPECIAL;
external modps is SPECIAL;
...
And finally the actual monthly monitoring rules in a new blacklist format: 
if ( OBSTYP = synop ) then
   if   VARIAB in ( z, ps )
   and  STATID = "    3ELC"
   then fail(); endif;

   if   VARIAB in ( z, ps, u10m, v10m )
   and  STATID = "   ELBX3"
   then fail(); endif;

return; endif;

if ( OBSTYP = airep ) then
   if  (VARIAB = t)
   and  STATID in ( "  N503US", "  UAL...")
   then fail(); endif;

return; endif;

if ( OBSTYP = satob ) then
   if   STATID in ( "     0//", "     024")
   then fail(); endif;

return; endif;

if ( OBSTYP = dribu ) then
   if   VARIAB in ( z, ps, u, v )
   and  STATID = "   46527"
   then fail(); endif;

return; endif;

if ( OBSTYP = temp ) then
   if  (VARIAB = z)
   and  STATID = "    ERES"
   then fail(); endif;

return; endif;

if ( OBSTYP = pilot ) then
   if   VARIAB in ( u, v )
   and  STATID = "   08221"
   then fail(); endif;

return; endif;

if ( OBSTYP = satem ) then
   if       STATID = "     201"
   then fail(); endif;

return; endif;
 
...
5.2 A more complex example
 
The Blacklist compiler will generate quite a compact and readable code from the following excerpt:
...
5.3 Adding completely new variable to the system
 
The current definition of variables can be checked from IFS source code in obs_preproc/blinit.F90. Adding new variables requires: 
Never remove or redifine existing variables. That will make re-running earlier cases virtually impossible.
Add the new variable in the SQL requests black_rob*.sql. If the new variable is not in hdr or body but in some data-specific tables (e.g. sat, or conv), you need to modify *only* those requests that are relevant for those data and have access to these tables.
Add a variable to the IFS source code in obs_preproc/blinit.F90.
Increase the number of defined variables in obs_preproc/blinit.F90.
External declaration must be done into the external-file.
Before starting to use the new variable, initialize it properly in obs_preproc/black.F90. If the new variable is not in hdr or body but in some data-specific tables (e.g. sat, or conv):make sure the variable is always initialized, and
put some logic in place (e.g. IF (IOBTYP == NSYNOP)...) in order to populate, only when appropriate, the variable with values from the sql.
The new variable can now be added into the blacklist. If keywords are associated with, declare them in the external-file as well.
...

Space shortcuts

Page tree

Versions Compared

Old Version 11

New Version 12

Key

2 The Blacklist language

2.1 Variables

A list of variables that are currently defined in IFS is given below. Adding new variables, see for 5.3.

2.1.1 Report characteristics

The up to date list of variables related to observation header and model fields can be found on HPCF in the external file of our blacklist (for instance /home/rd/rdx/data/37r3/an/external_bl_mon_monit.b for CY37R3).
...

2.1.2 Model/first guess characteristics

Variable Meaning Possible values
modps model surface pressure Real Value
modts model surface temperature Real Value
modt2m model 2 metre temperature Real Value
modtop model top level pressure (hPa) Real Value
sea_ice model sea-ice fraction Real Value
...

2.1.3 Observation characteristics

2.2 Keywords

Keywords are fixed values against which certain variables are compared. They should be consistent with the IFS definitions. A list of keywords that are currently defined in the blacklist (in the external file of our blacklist). Adding new keywords is straightforward.
...

2.3 Statements and operators

2.3.1 IF-statement syntax

2.3.3 List of more complex operators

Somewhat more complex operators can also be used to simplify coding. For example the compound AND-operators below:

2.4 Built-in functions

2.5 Actions

2.6 Variable declaration

3 Operational and experimental use of blacklist

3.1 Location of blacklist files

3.2 Some guidelines

4 Creating new blacklist file

4.1 Usage of the blcomp

4.2 Conversion from old to new blacklist

4.3 C-code generation

4.4 Linking with an application

4.5 Combining conversion and object generation

4.6 User interface

5 Examples

The blacklist file is normally about 1 000 lines long. In order not to confuse readers, we will explain here with very short examples what can be done with the blacklist

5.1 A simple example

5.2 A more complex example

5.3 Adding completely new variable to the system

Variable	Meaning	Possible values
modps	model surface pressure	Real Value
modts	model surface temperature	Real Value
modt2m	model 2 metre temperature	Real Value
modtop	model top level pressure (hPa)	Real Value
sea_ice	model sea-ice fraction	Real Value

Argument#1 (arg1)	Meaning
monthly	monthly monitoring (default)
constant	constant blacklisting
experimental	experimental blacklisting
use_emiskf_only	emiskf blacklisting

Argument#2 (arg2)	Meaning
level	Level of seriousness of blacklisting Range is between [0..1]. Default =1

Variable	Type	Meaning
NCMBLI	Integer	Blacklisting indicator 0= not blacklisted (default) 1= monthly monitoring 2= constant monitoring 3=experimental 4= use for emiskf only
ZCMCCC	Real	Seriousness of the blacklisting 0= Default if not blacklisted 1= Default if blacklisted (i.e. NCMBLI > 0) [0.01...0.99] for non-complete blacklisting (optional)
FEEDBACK	Integer	Feedback vector telling which variable(s) caused the blacklisting to occur: 0 = Blacklist line number where the fail()-function took action 1-N = Pointers to the variable indices to help to locate the responsible variables

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 11

New Version 12

Key

2 The Blacklist language

2.1 Variables

A list of variables that are currently defined in IFS is given below. Adding new variables, see for 5.3.

2.1.1 Report characteristics

The up to date list of variables related to observation header and model fields can be found on HPCF in the external file of our blacklist (for instance /home/rd/rdx/data/37r3/an/external_bl_mon_monit.b for CY37R3)....

2.1.2 Model/first guess characteristics

Variable Meaning Possible values modps model surface pressure Real Value modts model surface temperature Real Value modt2m model 2 metre temperature Real Value modtop model top level pressure (hPa) Real Value sea_ice model sea-ice fraction Real Value...

2.1.3 Observation characteristics

2.2 Keywords

Keywords are fixed values against which certain variables are compared. They should be consistent with the IFS definitions. A list of keywords that are currently defined in the blacklist (in the external file of our blacklist). Adding new keywords is straightforward....

2.3 Statements and operators

2.3.1 IF-statement syntax

2.3.3 List of more complex operators

Somewhat more complex operators can also be used to simplify coding. For example the compound AND-operators below:

2.4 Built-in functions

2.5 Actions

2.6 Variable declaration

3 Operational and experimental use of blacklist

3.1 Location of blacklist files

3.2 Some guidelines

4 Creating new blacklist file

4.1 Usage of the blcomp

4.2 Conversion from old to new blacklist

4.3 C-code generation

4.4 Linking with an application

4.5 Combining conversion and object generation

4.6 User interface

5 Examples

The blacklist file is normally about 1 000 lines long. In order not to confuse readers, we will explain here with very short examples what can be done with the blacklist

5.1 A simple example

5.2 A more complex example

5.3 Adding completely new variable to the system

The up to date list of variables related to observation header and model fields can be found on HPCF in the external file of our blacklist (for instance /home/rd/rdx/data/37r3/an/external_bl_mon_monit.b for CY37R3).
...

Variable Meaning Possible values
modps model surface pressure Real Value
modts model surface temperature Real Value
modt2m model 2 metre temperature Real Value
modtop model top level pressure (hPa) Real Value
sea_ice model sea-ice fraction Real Value
...

Keywords are fixed values against which certain variables are compared. They should be consistent with the IFS definitions. A list of keywords that are currently defined in the blacklist (in the external file of our blacklist). Adding new keywords is straightforward.
...