Page History

...

• Forecast Jump: When, for a given date, forecast results from sequential NWP model runs show a distinct and sudden change in forecast values (eg temperature forecasts valid at the same time for a given location might be 17ºC,16ºC,15ºC,22ºC ). 
• Forecasts Flip-flopping:  When forecast results from sequential NWP model runs alternate between higher and lower values (eg temperature forecasts valid at the same time for a given location might be 17ºC,13ºC,16ºC,12ºC,18ºC). 
• Forecast Trend:  When forecast results from sequential NWP model runs uniformly or otherwise move towards a lower or higher value (eg temperature forecasts valid at the same time for a given location might be 18ºC,16ºC,16ºC,15ºC,14ºC).

 Fig7Fig72.2.1A:  A representation of forecast temperatures at a certain location for a certain date produced by a series of forecast runs.  A jump may be considered as greater than some threshold δ.  Flip-flop may be considered as a sequence of results that alternately are higher or lower than its predecessor.  A trend may be considered as a sequence of results that rise or descend, uniformly or not.

...

Sometimes forecasts show significant and repetitive changes in predictions for a given location.  Often this is associated with the precise positioning of a trough or ridge in the vicinity of the location of interest (e.g. if in the Northern Hemisphere the axis lies to the east, then a northerly airflow brings colder temperatures and an associated type of weather; if to the west, then a southerly airflow brings warmer temperatures and a different weather type).  

Fig7Fig72.2.2B: An example of a forecast exhibiting jumpiness in the form of a major flip-flop.  On the diagram the y-axis is for forecast 2m temperatures, the x-axis shows the data time of the ensemble forecast.  The plotted values are for the forecast temperature at Paris verifying at 00Z 8 Dec 2016.  Forecast ensemble results are shown by box and whisker plots (described in Meteogram section), forecast ensemble mean values shown by black dots (red dots show values from the HRES).  Initially Day15 to Day11 forecasts were around 5°C or 6°C although with a broad range of up to ±8 to 10°C.   From 12UTC 27 Nov (Day10½) the forecast temperatures jumped to much colder values round -2°C with a relatively small spread of ±3 or4°C.  From 12UTC 30 Nov (Day7½) the forecast temperature rose suddenly back to around +6°C with a broader spread of ±8 to 10°C.  From 12UTC 3 Dec (Day4½) the forecast temperatures reverted to around +4°C with range of ±2 or 3C.   It should be remembered in general each ensemble solution should be viewed as one possible solution that is a member of a greater ensemble of the latest and recent solutions, although the later solutions do have the benefit of the most up to date data.

Fig7Fig72.2.3AC: An example of a forecast showing a major flip with a major difference in the forecast depth and extension of the upper trough over Eastern Europe.  The charts show 500hPa forecasts at 9km resolution VT 00UTC 25 Dec 2012 - upper chart: T+144 DT 00UTC 19 Dec 2012 with cold air across forecast Greece and SE Europe, lower chart: T+120 DT 00UTC 20 Dec 2012 with warm airmass over Greece and SE Europe.

Fig7Fig72.2.3BD: Standard ECMWF Mean and Spread charts for 850hPa temperature, verifying at 00UTC 25 Dec 2012, T+120 from ensemble control (CTRL) data time 00UTC 20 Dec 2012 (same case as in the lower panel of Fig7Fig72.2.3AC).

Left panel: The 500 hPa temperature forecast ensemble mean (isotherms) and normalized standard deviation (shading shows the normalised spread).  The normalised spread denotes how spread in the latest ensemble at a location compares with the ensemble spread at that location over recent (the last 30 days of) forecasts.  This chart, therefore, highlights uncertainty (green – relatively low, purple – relatively high).  So at Belgrade (blue circle) there is relatively high spread (uncertainty) and consequently one has less confidence in the forecast than might otherwise be expected for T+120 in that area.  Conversely the green area to the south of Ireland denotes less spread than seen in recent ensemble forecasts.

Right panel: The 500 hPa temperature forecast (at 9km resolution) and the actual ensemble Standard Deviation (at 18km resolution).  This shows that in absolute terms (as well as relative terms) there is a wide spread of ensemble solutions over eastern Europe (standard deviation between 4.5°C and 8.0°C).

Viewing Figure 7.2.3B alongside 7.2.3AFig72C alongside Fig72.D, one can find evidence for why the large jump in the forecast from a single ensemble member in the vicinity of Belgrade should not have come as such a surprise.  Uncertainty in absolute and relative terms was still very high after the jump.  So in broad terms the forecaster would be justified in following the jump, but at the same time should assign a large error bar to any issued forecasts.

...

• The proportion of previous forecasts that are "better" than the latest ones increases with lead-time:
  
  • at short lead-times a small but significant proportion appear better (~15% at Day2),
  • at longer lead-times a larger a larger proportion appear better (~40% at Day6).  (Fig7Fig72.2.4E).
• There is only a very small correlation between forecast jumpiness and the quality of the latest forecast (Fig7Fig72.2.5F).
  
• Beyond about Day3 the ensemble mean, by using results from all ensemble members, provides more consistent forecasts than the ensemble control.  This benefit gradually increases with forecast range.  
• The frequency of a flip (single jump) is very similar for both the ensemble mean and ensemble control.
• The frequency of flip-flopping occurs clearly less frequently in the ensemble mean than in the ensemble control.

Persson and Strauss (1995), Zsótér et al. (2009) found:

• the connection between forecast inconsistency (flip-flopping etc) and forecast error is weak,
• the average error of the ensemble mean relates quite strongly to the absolute spread in the ensemble.  
• on average, larger spread implies larger errors (this does not apply to the ensemble median or ensemble control, even if they happen to lie mid-range within the ensemble).

Fig7Fig72.2.4E:The likelihood that a forecast made 12hr or 24hr previously is “better” (in terms of RMSE) than the latest forecast.   The parameter is the MSLP for Northern Europe and the period October 2009-September 2010.  The result is almost identical if ACC is used as the verification measure. The diagram suggests that at longer lead-times (about Day6 or greater) about 40% of earlier forecasts are better than the latest though at short lead-times relatively few earlier forecasts are better (nevertheless 15%, or 1in6, appear better at T+48).

Fig7Fig72.2.5F:  The correlation of 24 hour forecast jumpiness and forecast error for 2m temperature against forecast lead-time for Heathrow at 12 UTC, October 2006 - March 2007.   At short lead-times the relationship between jumpiness and error is low, but increases with forecast range and asymptotically approaches 0.50 correlation.  Note that even for 0.5 the variance explained is still only 25%.

...

Table 772.2.1A:  The percentage of cases when >2mm/24hr has been observed when up to three consecutive ECMWF runs (T+84hr, T+96hr and T+108hr) have forecast >2mm/24hr for Volkel, Netherlands October 2007-September 2010.  R indicates where such rain has been forecast and has occurred.  Similar results are found for other west and north European locations and for other NWP medium-range models.

...

Table 7.2.2: Efficiency of intuitive techniques to assess outcomes given a series of forecast values from a series of sequential deterministic forecasts.

Fig7Fig72.2.6G: The graphs show sample schematic forecasts of 12UTC temperature over four successive NWP model runs: Jumpy (top) and Trend (bottom).  The histograms show the forecasts made by the students using their own techniques.   Spread was low with the jumpy forecast case since the oscillations remained fairly steady throughout, and the next forecast could be higher or lower without changing the range of the oscillation much. The spread was high with the trend forecast case illustrating the point that the next forecast may well be higher than the one before but destroy the trend, or lower than the one before, continuing the trend.

...

• adjust a forecast value (e.g. temperature, rainfall, etc) slightly lower or higher to follow the latest indications (e.g. warmer/cooler, wetter/drier, etc), but nevertheless to remain within the range of ensemble solutions from the latest and previous runs.  Reducing the change suggested by a noteworthy jump in the forecast can be the most appropriate course of action - but it does run the risk that the forecast from the next run will be even further away from the earlier solutions (i.e. the forecaster could be trying catch up with the NWP model forecasts and this illustrates one of the ways in which accuracy will be reduced).  On the other hand, it should be remembered that to follow a trend is also unreliable ~50% of the time.

• check whether the ensemble mean and probabilities are fairly consistent with previous runs.   If not, consider creating a lagged ensemble of the last two or three ensemble forecasts to give two or three times the number of members.  This will smooth out sudden changes in evolution while preserving the breadth of possible forecast extremes and probability information from the latest run.  A grand ensemble of ECMWF forecast results may be considered to compare latest forecast results with those of other state-of-the-art NWP models. 
  

• follow the ensemble mean rather than the ensemble control.  This can be more informative, especially at longer lead-times (say ≥ ~ 4 days).   However, note that strong gradients are always weakened in the ensemble mean and fine scale features (e.g. sting jets) will not be visible.
• inspect the Cumulative Density Function (CDF) of ensemble forecasts.  This can give a useful indication on the ensemble forecast values during the jumpiness.  At longer lead-times forecast CDFs may be similar to the M-climate.  But, with time, CDF between successive runs should show less lateral variation and tend to become steeper implying higher confidence.
  
  

Fig7Fig72.2.7H:An example of Cumulative Density Function (CDF) produced by a sequence of ensemble forecasts for precipitation at Zaga in Slovenia verifying for the 24hr 00UTC 27 to 00UTC 28 April 2017.  All show a very high extreme forecast index (EFI).  Note the four earlier CDFs (blues) showed a moderate slope indicating a spread of forecast precipitation intensities, and then jumped to a steeper slopes (purple and red) with lessening of spread of precipitation intensities.  Here the forecast showed a steady trend towards heavier precipitation with a jump to very heavy precipitation.  A forecaster would have been unwise at the time of the T+60 to 84hr forecast (rightmost dashed blue line) to think that this significantly wetter forecast overall was too much of  jump from the trend to be believed.

...