...

• Open your main Metview folder (select item 'Main Folder' from the Folders menu in the menu bar of the Metview Desktop)

• Open the subfolder called 'System' and then subfolder 'Defaults'.
• Do one of the following: edit the Symbol Plotting icon in the 'Defaults' folder to specify your new settings or else delete it and copy your 'symbol' icon into this folder then rename it 'Symbol Plotting'

...

In this exercise we will generate scatter plots for the analysis and first guess departures values of brightness temperature. As in the previous exercises we will use channel 5 from our 'AMSUA.odb' database. Please open folder 'scatter' inside folder 'odb_tutorial_4.1' to start the work.

About Scatter Plots

Scatter plots are used to display values in a Cartesian co-ordinate system for two variables for a set of data. In such plots the data is visualised as a collection of points where one variable specifies the positions on the horizontal axis and the other variable specifies the positions on the vertical axis. In our case column fg_depar@body defines the data for the horizontal axis and column an_depar@body defines the data for the vertical axis.

Creating an ODB Visualiser Icon

Create a new ODB Visualiser icon and rename it 'scatter_plot' then open its editor.
First, set ODB Plot Type to 'Xy Points' to indicate that we want to plot symbols in a Cartesian view.
Second, drop your 'AMSUA.odb' ODB Database icon into the Odb Data field. This specifies the database for which the query will be performed.
Last, we need to specify the ODB/SQL query and the way the columns are to be interpreted to generate the plot. We want to perform the following query: SELECT

SELECT 
 fg_depar@body,

...

 an_depar@body,

...

WHERE

...

 vertco_reference_1@body = 5 

In the ODB Visualiser interface this query cannot be typed in directly but has to be split into the following individual items:

• Odb X Variable: specifies the name of the column holding the x data in the SELECT statement (here fg_depar@body).
• Odb Y Variable: specifies the name of the column holding the y data in the SELECT statement (here an_depar@body).
• Odb Value Variable: specifies the name of the column whose values will be used to generate the colour palette for the scatter plot. We will leave this item empty because we do not want to use this feature in this example.
• Odb Where: specifies the WHERE statement. In our example it is as follows:

vertco_reference_1@body = 5 

Having finished the modifications your icon editor should look like this:

Running the Query

Save your ODB Visualiser icon (Apply) then right-click and execute to run the query. Within a few seconds the icon should turn green indicating that the retrieval was successful and has been cached. Now your icon behaves exactly like an ODB Database icon. Right-click examine to look at its content.

Visualising the Output

Right-click and visualise the icon to plot the data (please note that you can directly visualise this icon by skipping the execute step). This will bring up the Metview Display Window using the default visualisation (black circles) assigned to this kind of plots.


We can change the symbol (its type, colour and size) used for the plot with a Symbol Plotting icon. This time you do not need to create a new icon since there is one called 'scatter_symbol' already prepared for you. Edit this icon to see its settings then simply drag it into the Display Window and see how your plot has been changed.

Defining Binning

The main problem with our scatter plot is that it has dense regions where the data distribution is really hard to see. To overcome this difficulty we will create a density map out of our scatter plot. We can achieve it by turning our scattered dataset into a gridded dataset via binning. Binning means that we split the scatter plot area into grid cells by defining bins along the horizontal and vertical axes. Then for each cell we assign the number of points it contains as a grid value.
We will define the properties of the binning via the Binning icon.


Let's create a new Binning icon (it can be found in the Visual Definitions icon drawer, you may need to scroll the drawers to the right). Rename it 'bin_100' then edit it, by setting the following parameters:

...

With these settings we will split the data value range for both the x and y axes into 100 bins to generate the gridded dataset.
Now we will create a new ODB Visualiser icon to be used with our Binning icon. Copy your 'scatter_plot' icon (either right-click + duplicate, or drag with the middle mouse button), and rename the copy 'bin_plot' by clicking on its title. Edit the icon and make the following changes:
Set ODB Plot Type to 'Xy_Binning' to indicate that we want to generate a new dataset with binning and want to plot it in a Cartesian view.
Then, drop your 'bin_100' Binning icon into the Odb Binning field as the picture below illustrates it.

Visualising the Binned Dataset

Right-click and visualise icon 'bin_plot' to retrieve the data, perform the binning and plot the resulting dataset. This will bring up the Metview Display Window using the default contouring visualisation assigned to gridded datasets (your default contouring settings might be different to the one used to generate this plot).


Unfortunately isolines is not the desired visualisation type in our case (our data is not smooth enough) so we need to further customise the contouring settings. The best choice for us is to use grid shading since it applies shading for the grid cells themselves and we get the correct representation of our grid in the plot. (Please note that grid shading is different to cell shading, since the latter always involves an interpolation to define a new set of grid cells that the shading is applied for.)
Let's create a new Contouring icon (it can be found in the Visual Definitions icon drawer, you may need to scroll the drawers to the right).


Rename it 'bin_grid_shade' then edit it, by setting the following parameters:

...

Now drop this icon into the plot to see the effect of the changes.



In our Contouring icon we set the minimum value to '1.' to exclude grid cells containing no points at all and used 20 intervals between the minimum and the maximum to define the colour palette. Please note that our palette was automatically generated from a colour wheel by interpolating in clockwise direction between Contour Shade Min Level Colour and Contour Shade Max Level Colour.

Changing the View

We will further customise the plot by changing the axis value ranges and adding axis labels and grid-lines to it. To change these properties we need a Cartesian View icon (it can be found in the Visual Definitions icon drawer).


This time you do not need to create a new icon since there is one called 'scatter_view' already prepared for you. Edit this icon to see how the view is constructed (please note that the axis properties are defined via the embedded Horizontal Axis and Vertical Axis icons). Then simply drag it into the Display Window and see how your plot has been changed.

Changing the Title

To change the automatically generated ODB title you need to simply drag an already prepared Text Plotting icon called 'title' into the Display Window.

...

Plotting With Macro

In this example we will write the macro equivalent of the exercise we solved in Part 2 : we would like to retrieve and plot the brightness temperature for channel 5 for our 'AMSUA.odb' database. We will work in folder 'tb' again.

Basics

The implementation of ODB data plotting in Metview macro follows the same principles as in the interactive mode. In macro we work with the macro command equivalents of the ODB icons we have seen so far:

...

Another feature is that multi-line text, which we used for the WHERE statement in the ODB Visualiser icon, should be specified as a set of concatenated strings in macro. This technique is worth using if we do not want to put the (otherwise long) ODB queries into one line.

Automatic macro generation

The quickest way to generate a macro is to simply save a visualisation on screen as a Macro icon. Visualise your ODB data again, drop the symbol plotting and title icons into the plot and click on the macro icon in the tool bar of the Display Window.


Now a new Macro icon called 'MacroFrameworkN' is generated in your folder. Right-click visualise this icon. Now you should see your original plot reproduced.
Please note that this macro is to be used primarily as a framework. Depending on the complexity of the plot macros generated in this way may not work as expected and in such cases you may need to fine-tune them manually. So, we will use an alternative way and write our macro in the macro editor.

Step 1 - Writing a macro

Since we already have all the icons for our example we will not write the macro from scratch but instead we drop the icons into the Macro editor and just re-edit the automatically generated code.
Create a new Macro icon (it can be found in the Macros icon drawer) and rename it 'step1'. When you open the Macro editor (right-click edit) you can see that the first line contains #Metview Macro. Having this special comment in the first line helps Metview to identify the file as a macro, so we want to keep this comment in the first line.

Now position the cursor in the editor a few lines below the line of #Metview Macro. By doing so we specified the position where the icon-drop generated code will be placed. Then drop your 'tb_plot' ODB Visualiser icon into the Macro editor. You should see something like this (after removing the comment lines starting with # Importing):

#Metview Macro

...

 
 amsua_2e_db = read("AMSUA.odb")

...

 
 tb_plot = odb_visualiser(

...

 odb_where: " vertco_reference_1@body = 5 ",

...

 odb_data: amsua_2e_odb

...

 )

You only have to add the following command to the macro to plot the result:

plot(tb_plot) 

Now, if you execute this macro (right-click execute or click on the Play button in the Macro editor) you should see a Display Window popping up with your default symbol plotting visualisation.

Step 2 – Adding More Features

Duplicate the 'step1' Macro icon (right-click duplicate) and rename the duplicate 'step2'. In this step we will add our symbol plotting and title icons to the macro.
Position the cursor above the plot command in the Macro editor and drop your 'symbol_fixed' icon into it. Repeat this with the 'title' icon. Then modify the plot command by adding these new arguments after the tb_plot variable:

plot(tb_plot,symbol_fixed,title) 

Now, if you run this macro you should see your modified plot in the Display Window.
Remarks

• The macro equivalent of the wind plotting exercise in Part 3 and that of the scatter plot exercise in Part 4 can be written in a very similar way to what was shown above for symbol plotting. The solutions can be found and studied in folder 'wind_solution' and 'scatter_solution', respectively.

...

...

The ODB Filter Icon

In this exercise we will learn about the ODB Filter icon.


In the previous exercises we saw how to visualise ODB data with the ODB Visualiser icon. This icon is working well for visualisation, however, it is not suitable for performing general retrievals (with more than four columns) and does not allow direct access to the retrieved values (for data processing in macro). To achieve these goals we need to use the ODB Filter icon which is able to perform arbitrary ODB queries and save the results as new ODBs on output.

The Exercise

As a demonstration of the ODB Filter icon we will compute and plot the analysis increments in observation space for the brightness temperature. Like in Part 2 we will work again with channel 5 from our 'AMSUA.odb' database. Because our database does not contain the analysis increment we will compute it as the difference of the analysis departure and the first guess departure. First, we will write and perform the query needed for the exercise via the ODB Filter icon, then we will write a macro to do the computations and plot the final result.

The ODB Filter Icon

Open folder 'filter' inside your 'odb_tutorial_4.1' folder. Create a new ODB Filter icon (you can find it in the Filters icon drawer) and rename it 'filter_dep'. Open its editor and perform the following steps:

• Drop your 'AMSUA.odb' ODB Database icon into the Odb Data field. This specifies the database for which the query will be performed.
• Type in the following ODB/SQL query in the Odb Query multi-line text input field.

select

...

 
 lat@hdr as lat,

...

 lon@hdr as lon,

...

 fg_depar@body as fg_dep,

...

 an_depar@body as an_dep

...

where

...

 vertco_reference_1@body = 5 

Now your icon editor should look like this:


Remarks

• It is advised to use the full column names in the query because this fully complies with the SQL standards and newer versions of ODB require this as well.
• We used aliases (e.g. as lat) since it highly simplifies the referencing to the columns in the visualisation and macro processing.
• The ODB database for which the query is performed can be alternatively specified by the database path via the Odb Filename input field. Please note that the typed-in database path is only used by Metview if no database icon is present.
• The maximum number of rows accepted in the ODB retrieval is specified in the Odb Nb Rows input field. By default (-1) there is no upper limit for the number of rows.

...

Running the Query

Save your ODB Filter icon (Apply) then right-click and execute to run the query. Within a few seconds the icon should turn green indicating that the retrieval was successful and has been cached. Now your icon behaves exactly like an ODB Database icon. Right-click examine to look at its content.

Now you can see that as we defined it in the query the resulting ODB contains only four columns: lat, lon, fg_dep and an_dep.

Writing a Macro

Now we will write a macro to compute the analysis increment and visualise it.
Create a new Macro icon and rename it 'step1'. Open the Macro editor (right-click edit) and move the cursor somewhere below the #Metview Macro line at the top. Then drop your 'filter_dep' icon into the editor. You should see something like this (after removing the comment lines starting with # Importing):

#Metview Macro

...

 
 amsua_2e_odb= read("AMSUA.odb")

...

 
 filter_dep = odb_filter(

...

 odb_query: "select " &

...

 " lat@hdr as lat, " &

...

 " lon@hdr as lon, " &

...

 " fg_depar@body as fg_dep, " &

...

 " an_depar@body as an_dep " &

...

 "where " &

...

 " vertco_reference_1@body = 5 ",

...

 odb_data:amsua_2e_odb

...

 ) 

This piece of code performs the query and stores the result in an ODB database which is now represented by the filter_dep macro variable. Metview offers the values built-in macro function to read ODB column data into vectors. We need all the four columns for the computations and the visualisation, so we read them all one by one:

lat = values(filter_dep,"lat")

...

lon = values(filter_dep,"lon")

...

fg_dep = values(filter_dep,"fg_dep")

...

an_dep = values(filter_dep,"an_dep") 

Having each ODB column stored in a vector we compute the analysis increment as the difference between the analysis departure and the first guess departure. First, we allocate a vector to store the result:

num=count(lat)

...

incr = vector(num) 

then compute the difference between the all the elements of the an_dep and fg_dep vectors:

incr = an_dep –fg_dep 

The last step is the visualisation of the result. The simplest way is to build a geopoints object out of the needed vectors (these are lat, lon and incr, respectively) and pass it to the plot command:

geo=create_geo(num,"xyv")

...

 geo=set_latitudes(geo,lat)

...

 geo=set_longitudes(geo,lon)

...

 geo=set_values(geo,incr)

...

 
 plot(geo) 

Now, if you execute this macro (right-click execute or click on the Play button in the Macro editor) you should see a Display Window popping up with this result (your plot might look different depending on your default symbol plotting settings):

...

Improving the Plot

In our plot the large increments (in terms of absolute value) are not clearly highlighted because the plot is dominated by the bright green colour assigned to the near-zero values. To enhance the plot we would like to apply another colour palette by using:

...

It is complicated to create such a colour palette with one Symbol Plotting icon but we can overcome this difficulty by using three Symbol Plotting icons instead. This time you do not have to create these icons since they are already prepared for you.
Now visualise your plot again. Find the 'sym_small, 'sym_neg' and 'sym_pos' Symbol Plotting icons in the folder and select them together - either drag a rectangle around them, or click on each whilst holding down SHIFT. Then drag them together into the plot.

In the last step drag the 'title' Text Plotting icon into the plot, as well.
You will see an enhanced plot as shown below:

Modifying the Macro

We are satisfied with the new colour palette and with the title as well, so in the last step of the exercise we will add these new settings to our macro.
Duplicate the 'step1' macro icon (right-click Duplicate) and rename the duplicate 'step2'. Position the cursor above the plot command in the Macro editor and drop your 'sym_small', 'sym_neg', 'sym pos' and 'title' icons into it (you can drop them together or one by one).

Then modify the plot command by adding these new arguments after the geo variable:

plot(geo,sym_neg,sym_small,sym_pos,title) 

Now, if you visualise the macro you should see your modified plot in the Display Window.

...

...

Wind Plotting with ODB-2 Data

In this exercise we will present an example macro to show how to plot ODB-2 wind data if both the wind components are stored in the same column in the database. We will demonstrate this plotting technique on aircraft wind data above 250 hPa from our 'AIREP.odb' ODB-2 database.

Running the Macro

Open folder 'wind_odb2' inside your 'odb_tutorial_4.1' folder. You will find here the 'AIREP.odb' ODB Database icon and a macro called 'plot_wind'. If you right click and visualise the macro it will retrieve and plot the wind data from the database. You should see a Display Window popping up with the result (your plot might look different depending on your default wind plotting settings):

This result was achieved by executing the following steps in the macro:

• Perform two queries: one for the u and one for the v wind component.
• Get the data values from the two resulting ODBs as vectors.
• Build an xy_vector geopoints object out of this data
• Visualise the geopoints object

Explaining the Macro

Open the Macro editor for our macro (right-click edit) to study its source code.
The macro starts the #Metview Macro declaration. Then we load our database into a macro object.
 

mydb= read("AIREP.odb")

 
We perform two ODB/SQL queries for this database via the odb_filter command. The first query retrieves the u wind component while the second query retrieves the v wind component. As a result we have two ODB objects: filter_u containing columns lat, lon and u, and filter_v containing columns lat, lon and v. Since the u and v wind data come strictly after each other in the database we can be sure that the two ODBs have the same number of rows and their lat and lon columns are identical.

filter_u = odb_filter(

...

 odb_query: "select " &

...

 " lat@hdr as lat, " &

...

 " lon@hdr as lon, " &

...

 " obsvalue@body as u, " &

...

 "where " &

...

 " obstype@hdr = 2 and " &

...

 " varno@body = 3 and "

...

 " vertco_reference_1@body > 25000 ",

...

 odb_data:myodb

...

 ) 
 filter_v = odb_filter(

...

 odb_query: "select " &

...

 " lat@hdr as lat, " &

...

 " lon@hdr as lon, " &

...

 " obsvalue@body as v, " &

...

 "where " &

...

 " obstype@hdr = 2 and " &

...

 " varno@body = 4 and "

...

 " vertco_reference_1@body > 25000 ",

...

 odb_data:myodb

...

 ) 

In the next step we extract the lat, lon, u and v ODB columns into vectors via the values() macro function.

lat = values(filter_u,"lat")

...

 lon = values(filter_u,"lon")

...

 u = values(filter_u,"u")

...

 v = values(filter_v,"v")

Having each necessary ODB column stored in a vector we build an xy_vector geopoints object out of them.

geo=create_geo(count(lat),"xy_vector")

...

 geo=set_latitudes(geo,lat)

...

 geo=set_longitudes(geo,lon)

...

 geo=set_values(geo,u)

...

 geo=set_value2s(geo,v) 

In the last step we specify our wind plotting visual definition

colour_wind = mwind( … )

...

 
 then our title

...

 
 title = mtext( … ) 

and visualise the geopoints object itself

plot(geo,colour_wind,title) 

...

MARS Retrievals

In this exercise we will introduce how to retrieve ODB data from MARS in Metview.
Please note that the examples presented in this chapter are not guaranteed to work for you since the MARS ODB archive is still under development and subject to changes.

The MARS Retrieval Icon

In Metview we can access ODB from MARS by using the standard MARS Retrieval icon. This icon is located in the Data Acces icon drawer.


Now create a new MARS Retrieval icon by dragging it into your folder and rename it 'mars_hirs'. We will edit this icon in order to retrieve HIRS data available for yesterday at 00 UT and also use the filter option to select only a subset of the archived columns. Our retrieval can be written as follows:

retrieve,

...

 class = od,

...

 type= MFB,

...

 stream = DA,

...

 expver= 1,

...

 obsgroup= hirs,

...

 date = -1,

...

 time = 00,

...

 filter = "select lat,lon, obsvalue, vertco_reference_1" 

Now edit your MARS Retrieval icon so that it could perform this retrieval.
Remarks

• Please be aware that the Obsgroup parameter in the icon editor does not contain the string "hirs". Instead it offers a list of numerical IDs. The ID of HIRS is 2.

Running the Retrieval

Save your MARS Retrieval icon (Apply) then right-click and execute to run the query. Within a few seconds the icon should turn green indicating that the retrieval was successful and has been cached.

Working with the Retrieved Data

Now your icon behaves exactly like an ODB Database icon and all the relevant techniques introduced in the previous Chapters (Part 1 - Part 7 ) can be used with it. It means that you can examine, visualise and manipulate the data it holds.

• To examine: just right-click examine to look at its content.
• To visualise: you need to use an ODB Visualiser icon (see Part 2 ). There is one, called 'plot_hirs', already prepared for you in the folder. It is to plot the observation values for channel 5 on a map. Just edit this icon and drop your MARS Retrieval icon into the ODB Data field. Save the icon (Apply) then right-click and visualise to generate the plot.
• To manipulate: you need to write a macro (please read the next section about how to do it).

Macro Usage

The usage of ODB MARS retrievals in Metview macro follows the same principles as in the interactive mode. In macro we work with the macro command equivalent of the MARS Retrieval icon which is retrieve.
Now we will write a simple macro to retrieve our HIRS ODB data from MARS and compute and print the minimum, maximum and mean of the observed data values.
Create a new Macro icon and rename it 'step1'. Open the Macro editor (right-click edit) and move the cursor somewhere below the #Metview Macro line at the top. Then drop your 'mars_hirs' icon into the editor. You should see something like this (after removing the comment lines starting with # Importing):

#Metview Macro

...

 
 mars_hirs = retrieve(

...

 type : "mfb",

...

 repres : "bu",

...

 obsgroup : "hirs",

...

 time : 00,

...

 resol : "",

...

 filter : "select lat, lon, obsvalue, vertco_reference_1"

...

 ) 

This piece of code performs the retrieval and stores the result in an ODB database which is now represented by the mars_hirs macro variable.
In the next step we will use the values built-in macro function to read the ODB data values into a vector.

val = values(mars_hirs,"obsvalue@body") 

The last step is to compute some statistics for vector val and print them the into the standard output.

min_v=minvalue(val)

...

 max_v=maxvalue(val)

...

 mean_v=mean(val)

...

 print("min: ",min_v," max: ",max_v," mean: ",mean_v) 

Now, if you execute this macro (right-click execute or click on the Play button in the Macro editor) you should see the following text appearing in the standard output:
 

min: 191.710006714 max: 301.720001221 mean: 238.441305601

Further Macro Examples

There are two more macro examples in the folder to show you how to use ODB MARS retrievals in Metview macro:

• macro_filter: it shows how to manipulate ODB MARS data with the odb_filter function (see Part 6 ) to derive new datasets.
• macro_plot: it shows how to plot data from an ODB MARS retrieval. It is basically the macro equivalent of the 'plot_hirs' ODB Visualiser icon