Introduction

In these exercises we will look at a case study using a forecast ensemble. During the course of the exercise, we will explore the scientific rationale for using ensembles, how they are constructed and how ensemble forecasts can be visualised.

You will start by studying the evolution of the ECMWF HIRES forecast and ensemble forecast for this event. Then you will run your own OpenIFS forecast for a single ensemble member at lower resolutions and work in groups to study the OpenIFS ensemble forecasts.

Info

title	Caveat

In practise many cases are aggregated in order to evalute the forecast behaviour of the ensemble. However, it is always useful to complement such assessments with case studies of extreme events, like the one in this exercise, to get a more complete picture of IFS performance and identify weaker aspects that need further exploration.

Panel

title	St Jude wind-storm key highlights

The case study will look at one of several severe wind-storms that hit Europe in late 2013 (see handout of ECMWF article by Hewson et al, ECMWF Newsletter 139).

On the 28th October 2013 a small, severe wind-storm named St Jude in the UK, hit the UK & north-western Europe.
A total of 19 people were killed across Europe, 5 in the UK.
The return period of the event based on wind-gust observations show the 10yr return period was exceeded along the North Sea coast.

From the 23rd October, the ECMWF forecast predicted a greater than 70% probability of a severe wind event (greater than 60kt, 31m/s, at 1km) over southern England. A signal for the storm was evident from the 21st October.
On the 24th October, the UK MetOffice issued an amber alert for wind-speed across southern England placing the potential impact in the highest category.
The cyclone first appeared as a cold front wave, south of Nova Scotia late on 25th October.
It deepened and moved rapidly east then northeast, with the storm centre reaching southern Sweden late afternoon on the 28th.
The most rapid deepening occurred between 06-12UTC on the 28th between eastern England and the North Sea where the strongest wind gusts were observed.

Recap

Key points

Sources of uncertainty: initial analysis and model error.
Initial analysis uncertainty: sampled by use of Singular Vectors (SV) and Ensemble Data Assimilation (EDA).
Model uncertainty: sampled by use of stochastic processes. In IFS this means Stochastically Perturbed Physical Tendencies (SPPT) and the spectral backscatter scheme (SKEB)

ECMWF operational forecasts consist of:

HRES : T1279 (16km grid) highest resolution 10 day forecast
ENS : Ensemble (50 members), T639 for days 1-10, T319 days 11-15.

Exercise 1. Evaluating the ECMWF analyses and forecasts

Objective: Understanding the error in the forecast by comparing the ECMWF forecast with analysis & observations

Starting up metview

Code Block

title	Type the following command in a terminal window

metview

Info
Please enter the folder 'OpenIFS workshop 2015' to begin working.

Task 1: Visualise observations and ECMWF analyses

This task is done individually.

Note

icon	false
title	Key points

Examine windgust observations and ECMWF analyses. Note the observed area of strongest windgusts and their intensity.
Understand the storm development from the ECMWF analyses.

Panel

bgColor	white
titleBGColor	lightlightgrey
title	Metview icons

For this task, use the metview icons in the row labelled 'Analysis'

Panel

bgColor	white
titleBGColor	lightlightgrey
title	Plot wind gust observations

1. Right-click on the icon labelled 'wgust_obs.mv' and select 'Visualise' (or Control-I on the keyboard)

Panel

bgColor	white
titleBGColor	lightlightgrey
title	Plot analyses in various layouts

Icon 'an_1x1.mv' produces a single plot on the page.

Icon 'an_2x2.mv' can produce up to 4 plots per page.

Info

title	Change plot contents

For the 'an_1x1.mv' icon, the plot contents can be changed by editing the plot1 variable in the macro. By default the 10m wind is shown.

To alter the plotted field, right-click and choose 'Edit'.

It is possible to overlay multiple fields like this:

Image Added

You will find a list of available parameters in the macro.

After editing the macro text, you can optionally save using the 'File' menu and 'Save'.

Display the plot by clicking: Image Added

Task 1: Visualise operational forecast

Dates 24th - 29th October 2013.

Panel

Key questions:

Understand the synoptic situation and formation of the storm.
How does HIRES forecast compare to analysis and observations?

Suggested fields to plot: MSL, Z200, 10m wind and visualize the storm track.

Use the metview macros to plot different days and compare to analysis and plot forecast differences.

Panel

bgColor	white
title	Your notes:

Task 2 : Visualize the ensemble forecasts and ensemble spread

Again using the ECMWF operational forecast, look now at the 50 ensemble forecasts. These are at a lower resolution (T639) than the HIRES (T1279).

Objective: understand the forecast uncertainty

Panel

Key questions:

Visualise the ensemble mean - how does it compare to the HIRES forecast and analysis?
Visualize stamp map - are there any members that provide a better forecast?
Visualize spaghetti map - see how members spread over the duration of the forecast. How does the spread grow?

Suggested plots:

4 per frame: analysis, ensemble mean, spread and 1 other.
spaghetti plots (multiple plots per frame)
stamp plots.
Observations of wind gust + analysis (faked): 1 day only.

Note that as only 6hrly wind gust data is available from the operational forecasts, we have supplemented the 3hrly fields using forecast data.

Panel

bgColor	white
title	Your notes

Exercise 2. CDF/RMSE at different locations

Recap

TO DO: RMSE & CDF (concepts need explanation)

Objective: Understand ensemble reliability.

Choose 3 locations.e.g. Reading, Amsterdam, Copenhagen.
Using 10m wind and/or wind gust data plot CDF & RMSE curves using one of the OpenIFS forecasts and the ECMWF analysis data.
What is the difference and why?
Repeat for one of the other OpenIFS runs. Are there any differences, if so why?

Panel

bgColor	white
title	Your notes

Exercise 3. Creating an ensemble forecast using OpenIFS

See separate handout.

In this exercise, OpenIFS will be run on the ECMWF Cray XC30 to create a forecast for the storm at T319 resolution using only the stochastic schemes in the model. All forecasts are started from the same initial conditions based on the analysis.

Aim is to understand the impact of these different methods on the ensemble

Exercise 4. Verifying / Quantifying OpenIFS forecasts

Experiments available:

EDA+SV+SPPT+SKEB : Includes initial data and model uncertainty (experiment id : gbzl)
EDA+SV only : Includes only initial data uncertainty (experiment id: gc11)
SPPT+SKEB only : Includes model uncertainty only (run by participants, experiment id: oifs)

These are at T319 with start dates: 24/25/26/27 Oct 00Z for 5 days with 3hrly output.

Plots:

as above
4 frame: fc-an, fc-fc, pert.fc, ctl-an? *(compare the fc-fc maps with fc-an maps - can we see the uncertainty in the difference?)
PV maps

Task 1.

Objective: Understand the impact of changing the ensemble uncertainty

Look at ensemble mean and spread for all 3 cases.

How does it vary?
Which gives the better spread?
How does the forecast change with reducing lead time?

Task 2.

Look at different ensemble sizes?

Task 3.

Find an ensemble member that gives the best forecast and take the difference from the control. Step back to the beginning of the forecast and look to see where the difference originates from. How does this differ between the 3 OpenIFS runs? (with model uncertainty only, each initial state is identical so differences will develop from

Task 4.

Ensemble perturbations are applied in positive and negative pairs. For each perturbation computed, the initial fields are CNTL +/- PERT. (need a diagram here)

Choose an odd & even ensemble member from one of the 3 OpenIFS forecasts (e.g. members 9 and 10). For different forecast steps, compute difference of each member from the control forecast and then subtract those differences. (i.e. centre the differences about the control forecast).
What is the result? Do you get zero? If not why not? Use Z200 & Z500? MSLP?
Repeat looking at one of the other forecasts. How does it vary between the different forecasts?

If time:

Plot PV at 330K. What are the differences between the forecast? Upper tropospheric differences played a role in the development of this shallow fast moving cyclone.

Acknowledgements

Many people have contributed to this exercise and their contributions are acknowledged, in particular from ECMWF: Glenn Carver, Linus Magnusson, Martin Leutbecher, Sandor Kertesz, Iain Russell, Erland Kallen. From University of Oxford: Aneesh Subramanian, Peter Dueben, Peter Watson, Helen Christensen.

NOTES

These will disappear in the final handout.

Need to think how to group some of these activities so that people on a row are working together.

Panel

title	Data & plots required

MSLP
10m winds
T2m
Z500, Z200 winds geopotential.
wind gust : model & obs (nb. model wind gust data is accumulated)(Linus has windgust obs in geopoint format)
PV (decide which levels)
layout
contour / colourfill (overlay)
wind vector
spaghetti plots
stamp maps
rmse & cdf at several locations (user chooses)(Linus has macros for these)

Difference maps : to plot fc - an and ens member - control ( or ens member(i) - ensemble member(j) )
step animation of spaghetti plots etc to see spread developing.

Linus' vortex centre & tracking plot

Retrieve data from MARS for all apart from the OpenIFS experiment the participants will run themselves.

Note that operational ensemble runs at T319 are also available if we want to use them to compare 40r1 with OpenIFS? (see Linux for MARS script)

wind gust data is not simply max windspeed over 6hr period, it includes convective component (as well as conversion from windspeed to gust speed).

need to fake windgust data!

Panel

In exercise 1, the aim is to study the error between the forecast and the analysis.

Key points:

The forecast developed the storm too early, even for the forecast on the 27th. Storm was too far west.

plot Z200 to see position of the jetstream.

For the ensemble, plotting the mean will remove some features - might be useful for class to see this.

Note how spread is small for northerly low centre where uncertainty is much less.

Make sure they use 4 frame layout and plot 25th, 26th, 27th + analysis. Also, 25th, 26th & their difference

Panel

title	exercise

Question. How best to organise the experiments? Each user has an account or use one account with multiple directories?

Linus suggested running a script that reorders the data to have 1 file with all ensemble members for each field of interest. Do this after all members have run?

Panel

title	Comments from Erland

Suggests doing 2 more runs: EDA only and SV only. The perturbations from these have different characteristics and as we don't yet know what the results look like it might be useful to have these runs.

(I may not have all the details right - check with Linus who apparently did a PhD thesis on this)

Relate the perturbations to the baroclinic zones (ie. large scale flow). While the cyclone is developing the baroclinicity will be high and the EDA/SV perturbations will be more significant than when the storm is more developed with fronts when it will be more stable.

Should see that SV are quite artificial, some pertubations will grow rapidly and then die out. By contrast the perturbations in the EDA will be larger in the initial conditions.

Maybe we should plot the Eady index?

Erland says very important to plot these and see what we get in order to refine the questions and direction we want participants to take.

Panel

title	exercise 4

Linus explained that with the OpenIFS runs will have differing amounts of uncertainty, so the spread should noticeably change for points near the track in the analysis. This is particularly because of (a) timing error between the analysis & fc, (b) the ensemble tracks being more to the north of the analysis track. So Amsterdam for instance should see much less spread as the uncertainty in the ensemble is reduced.

Parameters that do not have a Gaussian like distribution in the ensemble can be problematic.

Space shortcuts

Page tree

Versions Compared

Old Version 23

New Version 24

Key

Introduction

Recap