3.2 Data quality

Data assimilation systems are based on methods that combine prior knowledge of the atmosphere (background) with observations in an optimal way taking into account statistical information about the errors of both pieces of information (Kalnay, 2003). Significant improvements in assimilation techniques and numerical weather forecasts in the 1980’s (short range forecast errors of similar magnitude as observation errors) allowed the use of data assimilation systems to provide diagnostic facilities to monitor the quality performance of the observational network (Hollingsworth, et al. 1986). The monitoring of data quality in WDQMS relies on the feedback from several NWP data assimilation systems - mainly the O-B departures. The quality/accuracy indicators to be considered are trueness, precision and gross error (WDQMS Guidance Document). However, for surface observations only trueness has been implemented in the web tool, whereas for upper-air observations both indicators (Trueness and Precision) are combined into a single accuracy metric.

Trueness 

The bias (an estimate of systematic error) is used as the measure of trueness (Table 3). The targets regarding trueness are stated so that the bias (average of O-B over a certain period) should be close to zero for all measured variables (sections 2.1 and 2.2). The trueness is assessed for all the temporal intervals considered in the tool (section 4): 6-hourly, daily and, in the future, monthly.  Also, a 5-day moving average (Alert) of the absolute value of daily calculated O-B (Table 9) needs to be calculated daily for all observed variables and compared against the prescribed thresholds (Table 6). This is used as one of the main performance indicators on the daily monitoring activities. 

Table 3 - Trueness

Precision 

The standard deviation (estimate of random error) is the quantitative measure of precision (Table 4). The targets for precision are applied to the standard deviation of O-B over a certain period for each of the observed variables (Table 6).  Like trueness, precision will be assessed 6-hourly, daily and monthly. Also, the 5-day moving average (Alert) of daily calculated standard deviation of O-B (Table 9) will be calculated for all variables and compared to the respective prescribed threshold ( Table 6).  This together with the performance indicator for Trueness will be used by the Evaluation function on their daily monitoring activities to determine the level of priority for stations showing accuracy/measurement uncertainty issues (see table in Annex2 of WDQMS Guidance Document). 

Table 4 - Precision

Both bias and standard deviation contribute to the overall measurement uncertainty. The root mean square error (rmse) is a common metric used to measure accuracy, and it is applied to upper-air observations in the WDQMS web tool (section 6.2) based on the quantitative information for the two vertical layers (mean and standard deviation of O-B departures over Trop and Stra, see section 2.2)  provided by the NWP reports (see Table 5).

Table 5 - Root Mean Square Error (rmse)

Note that quality indicators are applied only to the measured quantities whose O-B departures are available in the NWP monitoring reports, i.e. the ones whose model equivalent is available from the NWP assimilation system (see sections 2.1 and 2.2). Therefore, if the O-B departures are missing because the model background is not calculated in a particular NWP assimilation (e.g., not all centres compute O-B departures for observations they do not use in assimilation) the quality indicator will not be calculated and the station will not show up on the quality map. This is why some stations appear in the availability map, but not in the quality map. 

All measurement uncertainty targets are stated as targets for standard deviation (estimate of random errors) as a measure of observation precision. Biases should be avoided; that is, related targets are close to zero. Observations should be made in such a way that any biases (estimates of systematic errors) of measurement systems form only a small part of the measurement uncertainty. This is one of the reasons why monitoring the observation bias (Trueness) separately is crucial for the WDQMS web tool. Targets within WDQMS follow the WDQMS Guidance Document  (WMO, 2018), and consider mainly the use of “threshold” requirements (Table 6). However, considering the natural evolution of WDQMS and the tasks for the Regional WIGOS Centres (RWC), “breakthrough” requirements and “goals” are already introduced as descriptors (Table 7) in the quality maps (distinguished by different shades of green). The general definitions of these terms according to the WMO Rolling Review of Requirements (RRR) are:

• • “Threshold” is the minimum requirement to be met to ensure that data are useful;
• • “Goal” is an ideal requirement above which further improvements are not necessary;
• • “Breakthrough” is an intermediate level between “threshold” and “goal”, which, if achieved, would result in a significant improvement for the targeted application; the breakthrough level may be considered as optimal, from a cost–benefit point of view, when planning or designing observing systems.

Whenever the WDQMS performance targets are revised, the targets defined in WMO RRR for the global NWP application area are taken into account.  Table 7 contains the prescribed exceedance limits of measurement uncertainty for the variables monitored by WDQMS as defined by WMO RRR.

Table 6 - Exceedance thresholds of trueness and precision as defined in the WDQMS Guidance Document (WMO, 2018).

Table 7 - Exceedance limits of measurement uncertainty (combination of trueness and precision) for the relevant surface and upper-air variables as defined by WMO RRR (https://www.wmo-sat.info/oscar/requirements).