Observations used in the EQC activity

 CAMS is making use of a range of various observation types from different platforms and instruments:

• In-situ near-surface observations of air pollutants and greenhouse gases from monitoring stations or from ships. This includes surface concentrations of ozone, NO2 and PM2.5/PM10 relevant for air quality.
• Ground-based remote sensing data to observe concentrations above the surface, in the troposphere and stratosphere.
• Air-borne in-situ observations, e.g. measurements made with aircraft and balloons. 
• Satellite observations, in particular retrieval products that are not (yet) used in the assimilation.

The CAMS global products evaluated by the EQC activity are produced by combining state-of-the art modelling system with satellite observations, through a process called data assimilation. Here the model concentrations are optimised by using the satellite observations. The CAMS global EQC activity focuses especially on observations which are not used in the assimilation, in order to provide an independent assessment and uncertainty estimate of the CAMS global products. 

The observational datasets used for the evaluation of the CAMS global atmospheric composition forecasts are summarized in Table 1 below. A preference is given to routine observations provided by international networks (WMO-GAW, ICOS, NDACC, ACTRIS, AERONET, EEA, AirNow, CNEMC and others) which can guarantee high quality observations with standardised instrumentation, procedures and quality control.

A more detailed description of the observations used is available in the "Observations used for the validation of the CAMS global products" report, available from the global EQC reports page.

Figure 1: Observation types and major international networks providing measurement data.

Which CAMS products are evaluated?

The following CAMS global products are evaluated by the CAMS Global EQC activity:

Operational real-time CAMS global products

• CAMS global atmospheric composition forecasts
• CAMS global greenhouse gas forecasts. 

Reanalyses

• EAC4: The CAMS reanalysis of reactive gases and aerosols, covering the period 2003-January 2003 to August 2024.
• EGG4: The CAMS reanalysis for the greenhouse gases CO2 and CH4, covering the period 2003-2020.

System upgrades

The CAMS system is upgraded roughly once per year at the same time as the ECMWF IFS numerical weather prediction system, following the same cycle numbering. Recent upgrades are Cy49R1 (12 November 2024) and Cy48R1 (27 June 2023). Before an upgrade takes place the new system is operated in parallel to the operational system, and the performance of the new model cycle - evaluated with the observations mentioned above -  is compared with the performance of the old system. The results are summarised in a score card, see Figure 2. 

Control run

Apart from the analyses, CAMS is also producing a run without assimilating atmospheric composition satellite data, called the "control run".  This is used to evaluate the improvements brought by the assimilation. Note that the control run is not made available on the Atmosphere Data Store.

Figure 2: Score card for the CAMS upgrade to Cy49R1. Blue indicates an improved comparison against independent observations. Red indicates a degraded performance. On average the comparison against the observations showed a modest improvement for this upgrade.

Global EQC reports

The global forecasting system undergoes a constant assessment to ensure its results align with expectations. Regular Evaluation and Quality Assurance (EQA) reports are produced:

• The quality of the daily global analyses/forecasts is monitored in a series of 3-monthly reports for the meteorological seasons (DJF, MAM, JJA, SON)
• The comparison of the global reanalyses (EAC4, EGG4) against observations is reported each year covering the full time period, with a special focus on the stability of the time series.  
• The evaluation of planned system upgrades is summarised in a report which becomes available before the upgrade is implemented.

The validation reports contain:

• Detailed comparison results for the individual reactive trace gases, aerosol properties and greenhouse gases.
• A stand-alone summary of these results with the main conclusions.
• A set of detailed score boards, providing an overview of all comparisons. Separate score boards are produced comparing the analysis with the control run, comparing the analysis with the 4-day forecast, and comparing with the performance of the previous year. The score board is more extended than the score card shown in Figure 2 and is reporting various statistical scores for each measurement dataset.
• A set of case studies performed each quarter. These case studies are coordinated with the CAMS Weather Room activity.
• An overview of the CAMS system setup (modelling and data assimilation aspects) with references.
• The improvement of the system over time is also documented. This is linked to the CAMS global KPI's which are reported.

 In addition to these validation reports the activity is providing documentation

• detailing the observations and validation methods used, and
• detailing the scoring approaches used.

These reports are published and can be found on EQA reports of global services.

Figure 3: Validation report for the CAMS global atmospheric composition forecasts, covering the period June to August 2024. 

Validation servers

The Global EQC activity maintains two operational evaluation servers which provide public access to the comparison results: 

• The CAMS Global Validation Server visualizes the comparisons of the CAMS Near-Real-Time (NRT) products with observations. The server contains a very extended set of comparisons for more than 60 datasets, including surface in-situ, surface remote sensing, aircraft, balloon and satellite observations.  
• The CAMS Global EQC Server is focusing on long-term evaluation of the CAMS Reanalysis experiments (EAC4, EGG4), CAMS experiments, CAMS system upgrades as well as CAMS NRT forecasts. The server is responsive and has a powerful interface, providing customizable comparisons and summary statistics (e.g. heat maps).

Figure 4: Snapshot of the CAMS Global EQC Server