...

History of Modifications

...

Known Limitations of product

See section 3.4.

...

Known issues

...

1. Expand
   title Click here to expand the analysis about the rogue Tmin-24h values in AgERA5
   Background Several users have reported erroneous temperature values in the Tmin-24h variable where the value for selected grid cells could reach unrealistic values of around 220 K (-50 C) in locations with otherwise high temperatures. Analysis of the spatial distribution demonstrated that the cells with erroneous values can often be found in Western Australia but are not limited to that region and can be found in other parts of the World as well (Figure 1).  Image Added Image Added Image Added Image Added Figure 1: Maps of AgERA5 Tmin-24h with rogue values (black cells) for several regions in the World A further analysis on the occurrence of the rogue Tmin-24h values demonstrates that the problem occurs quite often. Figure 2 shows the number of files per year where such rogue values occur in the Tmin-24h variable. Note that files with rogue Tmin-24h values cannot be found by looking at the temperature extremes because the low Tmin-24h values are still within the valid range. E.g. an erroneous value of 220 K in Western Australia in Summer cannot be discriminated from a valid temperature value of 220K that occurs in Eastern Siberia at the same day. Instead, figure 2 was generated by computing the first order spatial differences and selecting on a threshold value. Surprisingly there are large differences between the different time-periods: the problem hardly occurs with the 1979-1999 time period, quite regularly in the period 2000-2020 and often since 2021. These time periods coincide with the batches in which the AgERA5 archive has been processed. The origin of the differences is not entirely clear but could be related to different encodings of the original ERA5 input data.  Image Added Figure 2: Number of Tmin-24 files per year where the problem of rogue values occurs Problem analysis To find the origin of the problem  it is needed to dive deep into the processing chain used for AgERA5 and the structure of the ERA5 files used as input for AgERA5. First of all, a feature of the ERA5 input files is that the content of each file does not contain the data for 00:00 to 24:00 UTC. For example, the ERA5 file containing air temperature data for 2024-01-01 contains data ranging from 2024-01-01T07:00:00 up till 2024-01-02T06:00:00. Therefore, the AgERA5 processing line first harmonizes all data files so they contain the time slices for the period 00:00 to 24:00 UTC. For example, for harmonizing the data for 2024-01-02 the processing line takes the files for 2024-01-01 and 2024-01-02, opens them jointly with xarray and takes the slice out of the dataset covering 2024-01-02T00:00:00 <= time < 2024-01-03T00:00:00. Analysis of the processing line of AgERA5 looking specifically what happens at those rogue Tmin-24h values demonstrated that the problem is generated at the step when two ERA5 values are joined (see Figure 3). Image Added Figure 3: Above: A timeseries of the original hourly ERA5 input data for variable MN2T (minimum temperature). Below: hourly ERA5 data clipped to 0-24UTC period. The grey area show the process of taking 2 ERA5 files and combining them into one new file during which the rogue Tmin-24h values are generated. A second point that should be understood is that figure 3 shows the ERA5 data as floating point values in degrees Kelvin. However, that is not how the data is stored on disk. The raw data coming from the ERA5 processing chain is not stored as floating point values (a 32 bit single-precision float) but instead as a C short datatype (a signed 16 bit integer) with an offset and a scaling factor associated with the variable as attributes. You can find out when looking at the data in Panoply. The AgERA5 Tmin-24h is derived from the ERA5 variable "mn2t" and panoply shows the scale_factor and offset values (figure 4).  Image Added Figure 4: Encoding of the variable mn2t in ERA5 input files. Tools like panoply and xarray handle this completely transparent on the background: they recognize the offset and scale_factor and convert back and forth. Moreover, the scale_factor and offset are highly optimized values: each ERA5 file has its own scale_factor and offset in order to maximize the precision for the given data range. The tricky part is when the newly sliced dataset has to be saved into a new NetCDF file which combines data from 2024-01-01 and 2024-01-02 (figure 3). Under the hood, xarray still knows that this data is represented by a C short with a scale_factor and offset, the question is now which scale_factor and offset to apply? The one for 2024-01-01 or the one for 2024-01-02? Xarray applies the scale_factor and offset from the first file it opens, so 2024-01-01 in this case. The location in time where things go wrong with the variable "mn2t" is marked with the square on the red curve in the figure 5 below. It is the first slice of the second input file (red line) and xarray is applying the scale_factor and offset of the first input file (green line) to save a new NetCDF file. Image Added Figure 5: As top figure 3, but with the input value that turns rogue marked with a black diamond. The temperature value that turns rogue is the first data point on the red curve (Figure 5) whose actual value is 318.871307 K and we convert it to 16 bit integer by inverting the scale and offset of the NetCDF file represented by the green curve (Figure 4): Code Block >>> math.trunc((318.871307 - 268.68162880225003)/0.0015210688706274414) 32996 And this is what happens with the last data point on the green curve whose values is 318.309113 K Code Block >>> math.trunc((318.309113 - 268.68162880225003)/0.0015210688706274414) 32626  But the maximum value that can be stored in a signed 16 bit integer is 32767. So the first data point on the red line (marked with the diamond) is too large to fit in the range represented by a 16-bit integer because the scale_factor and offset are not representative. The last data point on the green line just fits as it remains below 32767. Thus, the rogue Tmin-24h values come from an integer overflow case. Unfortunately, integer overflow does not generate any errors as it just rolls over towards the negative side of the signed integer (starting from -32768) so it is hard to detect. Fixing it in terms of software is relatively easy: we just have to force xarray to write NetCDF files with single precision floats instead of 16-bit integers. This takes twice as much disk space but these are only temporary files so that won't matter. Fixing it in terms of data is more tricky: a complete reprocessing of AgERA5 will be required. Since 10 March 2024, the processing line has been updated in order to avoid this problem. However, fixing the issue in the full AgERA5 dataset will require a reprocessing of the archive. We are currently investigating what the consequences are and if a full reprocessing is achievable. The consequences of the erroneous values for the fitness for purpose of the AgERA5 dataset are small. The cells with erroneous values are mostly located in deserts and other extremely warm areas which are usually not used for agriculture. Nevertheless such errors are undesirable and should preferably be fixed. Impact on AgERA5 variables All the input variables that are taken from ERA5 are stored as 16-bit C short datatypes and therefore the problem of integer overflow (and underflow!) could happen for any ERA5 input variable that is used for generating AgERA5. Nevertheless, the impact on different AgERA5 variables is different. Below there is a expert assessment on the impact of the different variables. Temperature Temperature variables that take the min or max of a time-slice can be affected directly depending on whether the rogue values is within the selection window (24h or day/night time) Assuming a maximum differences of 90 K for a rogue Temperature value, the temperature variables that are based on the mean are affected by a maximum of ~4 degrees K (90 K / 24 timesteps = 3.75 K) for 24h mean values or ~7.5 degrees K (90 K / 12 timesteps = 7.5 K) Precipitation Precipitation is affected slightly because the sum of all 24h values is taken. However, an overflow will turn a precipitation value for a single 1h time slice into a near-zero precipitation. The impact of this will not be noticeable due to the variable and erratic nature of precipitation Global radiation Global radiation is affected slightly. However, an overflow will turn a radiation value for a single 1h time slice into a near-zero radiation. The impact of this will not be noticeable due to the natural variability of radiation. Windspeed: Windspeed is hardly affected because an overflow or underflow will generate a windspeed in the opposite direction but at similar magnitude. The windspeed in AgERA5 is computed as the square root of the sum of the squared windspeeds in u and v direction. Therefore an over- or underflow will not cause a large difference in daily mean windspeed.  Humidity and vapour pressure: Individual humidity values could be affected when they coincide with a particular slice that is affected by rogue temperature values. Given that humidity is constrained between 0 and 100 % the impact is limited. Vapour pressure is computed as the mean of 24 timesteps and is therefore the impact is limited.  Snow thickness and LWE: Snow variables are calculated as the mean of 24 time slices and therefore the impact will be limited. Precipitation type: Precipitation type is based on a count of the different time steps and is therefore only in a limited degree affected.

Related articles

...