Introduction
For an operational service, it is crucial to provide timely and detailed quantitative information to the users on the uncertainty of the products. This information includes the evaluation of air pollutants such as ozone, CO and NO2, aerosol (optical properties, particulate matter), greenhouse gas concentrations (CO2, CH4) and UV radiation.
The CAMS Global EQC activity is comparing the global CAMS products with more than 65 independent measurement datasets.
The results of these evaluation activities are presented in validation reports and on evaluation web servers. These reports and servers can be accessed through the CAMS Global EQC page on the CAMS website: https://atmosphere.copernicus.eu/global-services
Observations used in the EQC activity
CAMS is making use of a range of various observation types from different platforms and instruments:
- In-situ near-surface observations of air pollutants and greenhouse gases from monitoring stations or from ships. This includes surface concentrations of ozone, NO2 and PM2.5/PM10 relevant for air quality.
- Ground-based remote sensing data to observe concentrations above the surface, in the troposphere and stratosphere.
- Air-borne in-situ observations, e.g. measurements made with aircraft and balloons.
- Satellite observations, in particular retrieval products that are not (yet) used in the assimilation.
The CAMS global products evaluated by the EQC activity are produced by combining state-of-the art modelling system with satellite observations, through a process called data assimilation. Here the model concentrations are optimised by using the satellite observations. The CAMS global EQC activity focuses especially on observations which are not used in the assimilation, in order to provide an independent assessment and uncertainty estimate of the CAMS global products.
The observational datasets used for the evaluation of the CAMS global atmospheric composition forecasts are summarized in Table 1 below. A preference is given to routine observations provided by international networks (WMO-GAW, ICOS, NDACC, ACTRIS, AERONET, EEA, AirNow, CNEMC and others) which can guarantee high quality observations with standardised instrumentation, procedures and quality control.
A more detailed description of the observations used is available in the "Observations used for the validation of the CAMS global products" report, available from the global EQC reports page.
Figure 1: Observation types and major international networks providing measurement data.
Table 1: Measurement datasets used in the global EQC activity.
| Network, dataset, instrument | Species, property observed |
|---|---|
Surface AQ network data (EEA, AirNow, CNEMC, other countries) | Surface O3, CO, NO2, SO2 |
Surface AQ network data (EEA, AirNow, CNEMC, other countries) | Surface PM2.5, PM10 |
WMO GAW, ESRL, IASOA | Surface O3, CO, NO2 |
| NDACC, FTIR | O3, NO2, HCHO, CO, CH4 (columns, subcolumns) |
| NDACC, lidar | O3 stratospheric profile |
| NDACC, Microwave Radiometer | O3 stratospheric profile |
| NDACC, Dobson | O3 total column |
NDACC, ZSL-DOAS | O3, NO2 stratospheric column, sunrise/sunset |
| Ozone sondes (NDACC, WOUDC, NILU, SHADOZ) | O3 profile (troposphere, stratosphere) |
| IAGOS, aircraft in-situ observations | O3, CO, NO2, NO, H2O, CO2, CH4 (profile, cruise data) |
| FRM4DOAS, PANDONIA; MAXDOAS & PANDORA | NO2, HCHO (tropospheric) column |
| ICOS, OBSPACK data collection, WDCGG | In-situ CO2, CH4, CO |
TCCON, COCCON | CO2, CH4, CO column |
| AirCore balloon | CO2 and CH4 vertical profile |
| UV stations | UV index, UV radiation |
| AERONET | AOD, AOD coarse/fine column, Ångström exponent |
ACTRIS lidar, E-Profile Ceilometer | Aerosol extinction |
GOME-2 A,B,C, TROPOMI | NO2, HCHO, SO2, CO columns |
IASI, MOPITT | O3, CO, SO2 columns |
ACE-FTS, Aura MLS, SAGE-III/ISS, OMPS-LP | O3, NOx, NOy, Cly, N2O, CH4, CFC-11 CFC-12 (stratospheric profiles) |
MODIS, VIIRS | AOD, Ångström exponent over ocean |
The evaluation of the CAMS global atmospheric composition forecasts is provided in quarterly reports. Only measurements which are available with a delay of less than one month behind real time can be used to evaluate the most recent CAMS results. This limits the datasets that can be used.
For the evaluation of the CAMS global reanalysis (EAC4 and EGG4) the set of observations is more extended because use can be made of observations which become available later, e.g. with a 1-2 year delay. One example is the use made of aircraft campaign data in the evaluation of EAC4.
One of the activities of the CAMS Global products EQC team is to further expand the number of observation datasets used for the validation. In particular, we are acquiring air-quality observations (PM2.5, PM10, ozone, NO2) from new networks worldwide. The three default networks are the European, North American and Chinese network. More recently the use of other data from South America and Asia (e.g. Chile, Taiwan and several others) has been added.
Which CAMS products are evaluated?
The following CAMS global products are evaluated by the CAMS Global EQC activity:
Operational real-time CAMS global products
Reanalyses
- EAC4: The CAMS reanalysis of reactive gases and aerosols, covering the period 2003-January 2003 to August 2024.
- EGG4: The CAMS reanalysis for the greenhouse gases CO2 and CH4, covering the period 2003-2020.
System upgrades
The CAMS system is upgraded roughly once per year at the same time as the ECMWF IFS numerical weather prediction system, following the same cycle numbering. Recent upgrades are Cy49R1 (12 November 2024) and Cy48R1 (27 June 2023). Before an upgrade takes place the new system is operated in parallel to the operational system, and the performance of the new model cycle - evaluated with the observations mentioned above - is compared with the performance of the old system. The results are summarised in a score card, see Figure 2.
Control run
Apart from the analyses, CAMS is also producing a run without assimilating atmospheric composition satellite data, called the "control run". This is used to evaluate the improvements brought by the assimilation. Note that the control run is not made available on the Atmosphere Data Store.
Figure 2: Score card for the CAMS upgrade to Cy49R1. Blue indicates an improved comparison against independent observations. Red indicates a degraded performance. On average the comparison against the observations showed a modest improvement for this upgrade.
Global EQC reports
The global forecasting system undergoes a constant assessment to ensure its results align with expectations. Regular Evaluation and Quality Assurance (EQA) reports are produced:
- The quality of the daily global analyses/forecasts is monitored in a series of 3-monthly reports for the meteorological seasons (DJF, MAM, JJA, SON)
- The comparison of the global reanalyses (EAC4, EGG4) against observations is reported each year covering the full time period, with a special focus on the stability of the time series.
- The evaluation of planned system upgrades is summarised in a report which becomes available before the upgrade is implemented.
The validation reports contain:
- Detailed comparison results for the individual reactive trace gases, aerosol properties and greenhouse gases.
- A stand-alone summary of these results with the main conclusions.
- A set of detailed score boards, providing an overview of all comparisons. Separate score boards are produced comparing the analysis with the control run, comparing the analysis with the 4-day forecast, and comparing with the performance of the previous year. The score board is more extended than the score card shown in Figure 2 and is reporting various statistical scores for each measurement dataset.
- A set of case studies performed each quarter. These case studies are coordinated with the CAMS Weather Room activity.
- An overview of the CAMS system setup (modelling and data assimilation aspects) with references.
- The improvement of the system over time is also documented. This is linked to the CAMS global KPI's which are reported.
In addition to these validation reports the activity is providing documentation
- detailing the observations and validation methods used, and
- detailing the scoring approaches used.
These reports are published and can be found on EQA reports of global services.
Figure 3: Validation report for the CAMS global atmospheric composition forecasts, covering the period June to August 2024.
Validation servers
The Global EQC activity maintains two operational evaluation servers which provide public access to the comparison results:
- The CAMS Global Validation Server visualizes the comparisons of the CAMS Near-Real-Time (NRT) products with observations. The server contains a very extended set of comparisons for more than 60 datasets, including surface in-situ, surface remote sensing, aircraft, balloon and satellite observations.
- The CAMS Global EQC Server is focusing on long-term evaluation of the CAMS Reanalysis experiments (EAC4, EGG4), CAMS experiments, CAMS system upgrades as well as CAMS NRT forecasts. The server is responsive and has a powerful interface, providing customizable comparisons and summary statistics (e.g. heat maps).
Figure 4: Snapshot of the CAMS Global EQC Server
References
A few key references for the CAMS global EQC activity are:
Eskes, H., et al., Technical note: Evaluation of the Copernicus Atmosphere Monitoring Service Cy48R1 upgrade of June 2023, Atmos. Chem. Phys., 24, 9475–9514, https://doi.org/10.5194/acp-24-9475-2024, 2024.
Agustí-Panareda, et al., Technical note: The CAMS greenhouse gas reanalysis from 2003 to 2020, Atmos. Chem. Phys., 23, 3829–3859, https://doi.org/10.5194/acp-23-3829-2023, 2023.
Wagner, A., et al., Comprehensive evaluation of the Copernicus Atmosphere Monitoring Service (CAMS) reanalysis against independent observations: Reactive gases. Elementa: Science of the Anthropocene (2021) 9 (1): 00171, https://doi.org/10.1525/elementa.2020.00171, 2021.
Wang, Y., Ma, Y.-F., Eskes, H., Inness, A., Flemming, J., and Brasseur, G. P., Evaluation of the CAMS global atmospheric trace gas reanalysis 2003–2016 using aircraft campaign observations, Atmos. Chem. Phys., 20, 4493–4521, https://doi.org/10.5194/acp-20-4493-2020, 2020.
Inness, A., et al., The CAMS reanalysis of atmospheric composition. Atmos.Chem. Phys. 2019, 19 (6), 3515-3556, https://doi.org/10.5194/acp-19-3515-2019.
Blechschmidt, A.-M., et al., Comparison of tropospheric NO2 columns from MAX-DOAS retrievals and regional air quality model simulations, Atmos. Chem. Phys., 20, 2795–2823, https://doi.org/10.5194/acp-20-2795-2020, 2020.
Akritidis, D., et al., A complex aerosol transport event over Europe during the 2017 Storm Ophelia in CAMS forecast systems: analysis and evaluation, Atmos. Chem. Phys., 20, 13557–13578, https://doi.org/10.5194/acp-20-13557-2020, 2020.
