...

Available plot types

...

Task 1:

...

This is similar to task 1 in exercise 2, except the RMSE curves for all the ensemble members from a particular forecast will be plotted.

Right-click the ens_rmse.mv icon, select 'Edit' and plot the curves for 'mslp' and 'z500'.

Change 'expID' for your choice of ensemble.

clustersId="off"

Clustering will be used in later tasks.

There might be some evidence of clustering in the ensemble plumes.

There might be some individual forecasts that give a lower RMS error than the control forecast.

If time:

• Explore the plumes from other variables.
• Do you see the same amount of spread in RMSE from other pressure levels in the atmosphere?

Task 2: Ensemble spread

In the previous task, uncertainty in the forecast by starting from different initial conditions and the stochastic parameterizations can result in significant differences in the RMSE (for this particular case and geographical region).

This task will explore the difference in another way by looking at the 'ensemble spread'.

Use the ens_to_an.mv icon and plot the MSLP and z500. This will produce plots showing: the mean of  all the ensemble forecasts, the spread of the ensemble forecasts, the operational HRES deterministic forecast and the analysis.

Change 'expId' if required.

Animate this plot to see how the spread grows.

This macro can also be used to look at clusters of ensemble members. It will be used later in the clustering tasks. For this task, make sure all the members of the ensemble are used.

#ENS members (use ["all"] or a list of members like [1,2,3]
members=["all"]        #[1,2,3,4,5] or ["all"] or ["cl.example.1"]

Task 3: Spaghetti plots - another way to visualise spread

A "spaghetti" plot is where a single contour of a parameter is plotted for all ensemble members. It is another way of visualizing the differences between the ensemble members and focussing on features.

Use the ens_to_an_runs_spag.mv icon. Plot and animate the MSLP and z500 fields using your suitable choice for the contour level. Find a value that highlights the low pressure centres. Note that not all members may reach the low pressure set by the contour.

The red contour line shows the control forecast of the ensemble.

Note that this macro may animate slowly because of the computations required.

Experiment with changing the contour value and (if time) plotting other fields.

Task 4: Visualise ensemble members and differences

So far we have been looking at reducing the information in some way to visualise the ensemble.

Stamp maps are used to visualise all the ensemble members as normal maps. These are small, stamp sized contour maps plotted for each ensemble member using a small set of contours.

There are two icons to use, stamp.mv and stamp_diff.mv.

Use stamp.mv to plot the MSLP and z500 fields in the ensemble.

The stamp map is slow to plot as it reads a lot of data. Rather than animate each forecast step, a particular date can be set by changing the 'steps' variable.

#Define forecast steps
steps=[2012-09-24 00:00,"to",2012-09-24 00:00,"by",6]

Make sure clustersId="off" for this task.

Precipitation over France

Use stamp.mv and plot total precipitation ('tp') over France (mapType=2) for 00Z 24-09-2012 (compare with Figure 2 in Pantillon).

Note, stamp_diff.mv cannot be used for 'tp' as there is no precipitation data in the analyses.

Difference stamp maps

Use the stamp_diff.mv plot to look at the differences between the ensemble members and the analysis. It can be easier to understand the difference in the ensembles by using difference stamp maps.

Compare ensemble members to analysis

After visualizing the stamp maps, it can be useful to animate a comparison of individual ensemble members to the analyses.

ens_to_an_diff.mv and pf_to_cf_diff.mv can be used to compare ensemble members.

...

 To animate the difference in MSLP of an individual ensemble member 30 to the analysis, edit the lines:

param="mslp"
ensType="pf30"

To compare the control forecast:

ensType="cf"

Further analysis using ensembles

...

This will show the forecasts from the ensemble members and also their difference with the ensemble control forecast.

To animate the difference in MSLP with ensemble members '30' and '50', set:

param="mslp"
pf=[30,50]

...

Find ensemble members that appear to produce a better forecast and look to see how the initial development in these members differs.

• Select 'better' forecasts using the stamp plots and use ens_to_an.mv to modify the list of ensemble plots.
• Use pf_to_cf_diff and ens_to_an_diff to take the difference between these perturbed ensemble member forecasts from the control and analyses to also look at this.

Ensemble spread

This task will explore the 'ensemble spread'.

Use the ens_to_an.mv icon and plot the MSLP and z500. This will produce plots showing: the mean of  all the ensemble forecasts, the spread of the ensemble forecasts and the operational HRES deterministic forecast.

Change 'expId' if required to select either the 2012 ensemble expId="ens_oper" or the reforecast ensemble expId="ens_2016".

Animate this plot to see how the spread grows.

This macro can also be used to look at clusters of ensemble members. It will be used later in the clustering tasks. For this task, make sure all the members of the ensemble are used.

#ENS members (use ["all"] or a list of members like [1,2,3]
members=["all"]        #[1,2,3,4,5] or ["all"] or ["cl.example.1"]

Task 3: Spaghetti plots - another way to visualise spread

A "spaghetti" plot is where a single contour of a parameter is plotted for all ensemble members. It is another way of visualizing the differences between the ensemble members and focussing on features.

Use the ens_to_an_runs_spag.mv icon. Plot and animate the MSLP and z500 fields using your suitable choice for the contour level. Find a value that highlights the low pressure centres. Note that not all members may reach the low pressure set by the contour.

The red contour line shows the control forecast of the ensemble.

Note that this macro may animate slowly because of the computations required.

Experiment with changing the contour value and (if time) plotting other fields.

Task 4: Visualise ensemble members and differences

So far we have been looking at reducing the information in some way to visualise the ensemble.

Stamp maps are used to visualise all the ensemble members as normal maps. These are small, stamp sized contour maps plotted for each ensemble member using a small set of contours.

There are two icons to use, stamp.mv and stamp_diff.mv.

Use stamp.mv to plot the MSLP and z500 fields in the ensemble.

The stamp map is slow to plot as it reads a lot of data. Rather than animate each forecast step, a particular date can be set by changing the 'steps' variable.

#Define forecast steps
steps=[2012-09-24 00:00,"to",2012-09-24 00:00,"by",6]

Make sure clustersId="off" for this task.

Precipitation over France

Use stamp.mv and plot total precipitation ('tp') over France (mapType=2) for 00Z 24-09-2012 (compare with Figure 2 in Pantillon).

Note, stamp_diff.mv cannot be used for 'tp' as there is no precipitation data in the analyses.

Difference stamp maps

Use the stamp_diff.mv plot to look at the differences between the ensemble members and the analysis. It can be easier to understand the difference in the ensembles by using difference stamp maps.

Compare ensemble members to analysis

After visualizing the stamp maps, it can be useful to animate a comparison of individual ensemble members to the analyses.

ens_to_an_diff.mv and pf_to_cf_diff.mv can be used to compare ensemble members.

param="mslp"
ensType="pf30"

ensType="cf"

Further analysis using ensembles

param="mslp"
pf=[30,50]

Task 5:  Cumulative distribution function

Recap

Cumulative distribution function (CDF)

The figures above illustrate the relationship between a normal distribution and its associated cumulative distribution function. The CDF is constructed from the area under the probability density function.

The CDF gives the probability that a value on the curve will be found to have a value less than or equal to the corresponding value on the x-axis. For example, in the figure, the probability for values less than or equal to X=0 is 50%.

The shape of the CDF curve is related to the shape of the normal distribution. The width of the CDF curve is directly related to the value of the standard deviation of the probability distribution function.

For an ensemble, the width is therefore related to the 'ensemble spread'.

For a forecast ensemble where all values were the same, the CDF would be a vertical straight line.

Plot the CDFs

Image Added

This exercise uses the cdf.mv icon. Right-click, select 'Edit' and then:

• Plot the CDF of MSLP for Toulouse for your choice of ensemble
• Find a latitude/longitude point in the area of intense precipitation on 12Z 24/9/2012 (see Figure 2(c) Pantillon et al) and plot the CDF for MSLP (set station=[lat,lon] in the macro cdf.mv)

Note that only MSLP, 2m temperature (t2) and 10m wind-speed (speed10) are available for the CDF.

Make sure useClusters='off'.

Exercise 4: Cluster analysis

The paper by Pantillon et al, describes the use of clustering to identify the main scenarios among the ensemble members.

This exercise repeats some of the plots from the previous one but this time with clustering enabled.

Using clustering will highlight the ensemble members in each cluster in the plots.

In this exercise you will:

• Construct your own qualitative clusters by choosing members for two clusters
• Generate clusters using principal component analysis (similar to Pantillon et al).

Task 1: Create your own clusters

Clusters can be created manually from lists of the ensemble members.

Choose members for two clusters. The stamp maps are useful for this task.

From the stamp map of z500 at 24/9/2012 (t+96), identify ensemble members that represent the two most likely forecast scenarios.

It is usual to create clusters from z500 as it represents the large-scale flow and is not a noisy field. However, for this particular case study, the stamp map of 'tp' (total precipitation) over France is also very indicative of the distinct forecast scenarios.

1#   2  3  4  9  22 33 40
2#   10 11 12 31 49

Task 5:  Cumulative distribution function

Recap

Cumulative distribution function (CDF)

The figures above illustrate the relationship between a normal distribution and its associated cumulative distribution function. The CDF is constructed from the area under the probability density function.

The CDF gives the probability that a value on the curve will be found to have a value less than or equal to the corresponding value on the x-axis. For example, in the figure, the probability for values less than or equal to X=0 is 50%.

The shape of the CDF curve is related to the shape of the normal distribution. The width of the CDF curve is directly related to the value of the standard deviation of the probability distribution function.

For an ensemble, the width is therefore related to the 'ensemble spread'.

For a forecast ensemble where all values were the same, the CDF would be a vertical straight line.

Plot the CDFs

Image Removed

This exercise uses the cdf.mv icon. Right-click, select 'Edit' and then:

• Plot the CDF of MSLP for Toulouse for your choice of ensemble
• Find a latitude/longitude point in the area of intense precipitation on 12Z 24/9/2012 (see Figure 2(c) Pantillon et al) and plot the CDF for MSLP (set station=[lat,lon] in the macro cdf.mv)

Note that only MSLP, 2m temperature (t2) and 10m wind-speed (speed10) are available for the CDF.

Make sure useClusters='off'.

Exercise 4: Cluster analysis

The paper by Pantillon et al, describes the use of clustering to identify the main scenarios among the ensemble members.

This exercise repeats some of the plots from the previous one but this time with clustering enabled.

Using clustering will highlight the ensemble members in each cluster in the plots.

In this exercise you will:

• Construct your own qualitative clusters by choosing members for two clusters
• Generate clusters using principal component analysis (similar to Pantillon et al).

Task 1: Create your own clusters

Clusters can be created manually from lists of the ensemble members.

Choose members for two clusters. The stamp maps are useful for this task.

From the stamp map of z500 at 24/9/2012 (t+96), identify ensemble members that represent the two most likely forecast scenarios.

It is usual to create clusters from z500 as it represents the large-scale flow and is not a noisy field. However, for this particular case study, the stamp map of 'tp' (total precipitation) over France is also very indicative of the distinct forecast scenarios.

#ENS1# members  2  3  4  9  22 33 40
2#   10 11 12 31 49

(use ["all"] or a list of members like [1,2,3]
members_1=["cl.example.1"]
members_2=["cl.example.2"]

Task 2: Empirical orthogonal functions / Principal component analysis

A quantitative way of clustering an ensemble is by computing empirical orthogonal functions from the differences between the ensemble members and the control forecast.

Although geopotential height at 500hPa at 00 24/9/2012 is used in the paper by Pantillon et al., the steps described below can be used for any parameter at any step.

Image Added

The eof.mv macro computes the EOFs and the clustering.

param="z500"
expId="ens_oper"
steps=[2012-09-24 00:00]

ens_oper_cluster.eof.txt

#ENS members (use ["all"] or a list of members like [1,2,3]
members_1=["cl.example.1"]
members_2=["cl.example.2"]

Task 2: Empirical orthogonal functions / Principal component analysis

A quantitative way of clustering an ensemble is by computing empirical orthogonal functions from the differences between the ensemble members and the control forecast.

Although geopotential height at 500hPa at 00 24/9/2012 is used in the paper by Pantillon et al., the steps described below can be used for any parameter at any step.

Image Removed

The eof.mv macro computes the EOFs and the clustering.

param="z500"
expId="ens_oper"
steps=[2012-09-24 00:00]

ens_oper_cluster.eof.txt

#ENS members (use ["all"] or a list of members like [1,2,3]
members_1=["cl.eof.1"]
members_2=["cl.eof.2"]

#ENS members (use ["all"] or a list of members like [1,2,3]
members_1=["cl.eof.1"]
members_2=["cl.eof.2"]

Exercise 5. Percentiles and probabilities

To further compare the 2012 and 2016 ensemble forecasts, plots showing the percentile amount and probabilities above a threshold can be made for total precipitation.

Use these icons:

Image Added

Both these macros will use the 6-hourly total precipitation for forecast steps at 90, 96 and 102 hours, plotted over France.

Task 1. Plot percentiles of total precipitation

Edit the percentile_tp_compare.mv icon.

Set the percentile for the total precipitation to 75%:

#The percentile of ENS precipitation forecast
perc=75

Run the macro and compare the percentiles from both the forecasts.  Change the percentiles to see how the forecasts differ.

Task 2: Plot probabilities of total precipitation

This macro will produce maps showing the probability of 6-hourly precipitation for the same area as in Task 1.

In this case, the maps show the probability that total precipitation exceeds a threshold expressed in mm.

Edit the prob_tp_compare.mv and set the probability to 20mm:

#The probability of precipitation greater than
prob=20

Run the macro and view the map. Try changing the threshold value and run.

Exercise 6. Assessment of forecast errors

In this exercise, various methods for presenting the forecast error are presented.

Task 1: Forecast error

In this task, we'll look at the difference between the forecast and the analysis by using "root-mean-square error" (RMSE) curves as a way of summarising the performance of the forecast.

Root-mean square error curves are a standard measure to determine forecast error compared to the analysis and several of the exercises will use them. The RMSE is computed by taking the square-root of the mean of the forecast difference between the HRES and analyses. RMSE of the 500hPa geopotential is a standard measure for assessing forecast model performance at ECMWF (for more information see: http://www.ecmwf.int/en/forecasts/quality-our-forecasts).

Right-click the hres_rmse.mv icon, select 'Edit' and plot the RMSE curve for z500.

Repeat for the mean-sea-level pressure mslp.

Repeat for both geographical regions: mapType=1 (Atlantic) and mapType=2 (France).

Task 2: Compare forecast to analysis

Use the hres_to_an_diff.mv icon and plot the difference map between the HRES forecast and the analysis for z500 and mslp.

If time: look at other fields to study the behaviour of  the forecast.

Task 3: RMSE "plumes" for the ensemble

This is similar to task 1 in exercise 2, except the RMSE curves for all the ensemble members from a particular forecast will be plotted.

Right-click theens

Exercise 5. Percentiles and probabilities

To further compare the 2012 and 2016 ensemble forecasts, plots showing the percentile amount and probabilities above a threshold can be made for total precipitation.

Use these icons:

Image Removed

Both these macros will use the 6-hourly total precipitation for forecast steps at 90, 96 and 102 hours, plotted over France.

Task 1. Plot percentiles of total precipitation

Edit the percentile_tp_compare.mv icon.

Set the percentile for the total precipitation to 75%:

#The percentile of ENS precipitation forecast
perc=75

Run the macro and compare the percentiles from both the forecasts.  Change the percentiles to see how the forecasts differ.

Task 2: Plot probabilities of total precipitation

This macro will produce maps showing the probability of 6-hourly precipitation for the same area as in Task 1.

In this case, the maps show the probability that total precipitation exceeds a threshold expressed in mm.

Edit the prob_tp_compare.mv and set the probability to 20mm:

#The probability of precipitation greater than
prob=20

Run the macro and view the map. Try changing the threshold value and run.

Exercise 6. Assessment of forecast errors

In this exercise, various methods for presenting the forecast error are presented.

Task 1: Forecast error

In this task, we'll look at the difference between the forecast and the analysis by using "root-mean-square error" (RMSE) curves as a way of summarising the performance of the forecast.

Root-mean square error curves are a standard measure to determine forecast error compared to the analysis and several of the exercises will use them. The RMSE is computed by taking the square-root of the mean of the forecast difference between the HRES and analyses. RMSE of the 500hPa geopotential is a standard measure for assessing forecast model performance at ECMWF (for more information see: http://www.ecmwf.int/en/forecasts/quality-our-forecasts).

Right-click the hres_rmse.mv icon, select 'Edit' and plot the RMSE curve for z500.

Repeat for the mean-sea-level pressure mslp.

curves for 'mslp' and 'z500'.

Change 'expID' for your choice of ensemble.

clustersId="off"

Clustering will be used in later tasksRepeat for both geographical regions: mapType=1 (Atlantic) and mapType=2 (France).

Task 2: Compare forecast to analysis

Use the hres_to_an_diff.mv icon and plot the difference map between the HRES forecast and the analysis for z500 and mslp.

...

There might be some evidence of clustering in the ensemble plumes.

There might be some individual forecasts that give a lower RMS error than the control forecast.

If time:

• Explore the plumes from other variables.
• Do you see the same amount of spread in RMSE from other pressure levels in the atmosphere?

Appendix

Further reading

For more information on the stochastic physics scheme in (Open)IFS, see the article:

...