Contributors: A. Velazquez Blazquez (Royal Meteorological Institute of Belgium (RMIB)), N. Clerbaux (RMIB), E. Baudrez (RMIB)
Issued by: RMIB
Date: 08/07/2021
Ref: C3S_D312b_Lot1.2.5.5-v1.2_202103_PQAD_ECVEarthRadiationBudget_v1.2
Official reference number service contract: 2018/C3S_312b_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
List of tables
List of figures
Scope of the document
This Product Quality Assurance Document provides a description of the whole product validation methodology for the HIRS Outgoing Longwave Radiation (OLR) Climate Data Record (CDR). This CDR is "brokered" into the Climate Data Store (CDS) from National Oceanic and Atmospheric Administration (NOAA) / National Centers for Environmental Information (NCEI). Therefore, this document only provides links to the relevant validation documents and published papers.
Executive summary
The NOAA/NCEI, the producer of the HIRS OLR data, provides the following executive summary for the v2.7 of the CDR:
This Climate Data Record (CDR) of monthly mean Outgoing Longwave Radiation (OLR) flux at the top of the atmosphere in all sky conditions is on a 2.5 degree x 2.5 degree grid with global coverage from January 1979 to the present and continuing monthly. It is derived based on the multispectral High Resolution Infrared Radiation Sounder (HIRS) OLR algorithm and other methodologies. This dataset was created from HIRS OLR retrievals from TIROS-N to NOAA-19, and MetOp-A/B satellites with inter-satellite calibration adjustments and employed the empirical diurnal model for monthly mean derivation. This version 2.7 is an upgrade from the previous version 2.2-1. The main change is in the OLR regression models and the inter-satellite calibration. The version 2.7 OLR regression models improved the retrieval consistency between variant versions of HIRS instruments, HIRS-2, 2i, 3 and 4, and in turn, the accuracy of inter-satellite calibration is significantly improved, thus eliminating the spurious trends presented in the version 2.2-1 data. The data file format is netCDF-4 with CF metadata, and it is accompanied by algorithm documentation, data flow diagram and source code for the NOAA CDR Program.
This version (v02r07 or simply "2.7") of the OLR-monthly CDR has been released on 31/08/2018, and new data have been added since then. Now, it contains more than 40 years of satellite-based measurements of the OLR, on a global scale and at 2.5° x 2.5° spatial resolution. Table 1 provides a summary of the accuracies for the HIRS OLR CDR.
Table 1: Summary of the accuracy of the monthly mean OLR at 2.5° x 2.5° resolution provided in v02r07 of the CDR. The uncertainty is provided at "1 standard deviation" from the intercomparison with CERES EBAF Edition 4.0 (see section 4.1).
Product Name | Processing uncertainties |
OLR - Outgoing Longwave Radiation | 1.8 W/m² |
1. Validated products
The version v02.r07 of the HIRS OLR CDR provides monthly mean Outgoing Longwave Radiation (OLR) in all sky conditions. The dataset comprises 42 years and 5 months (January 1979 – Feb 2021) of satellite-based measurements derived from the High-Resolution Infrared Radiation Sounder (HIRS) instruments onboard the polar orbiting NOAA and MetOp satellites.
The HIRS OLR CDR is a level 3 product (monthly means) on a regular global latitude-longitude grid with 2.5° x 2.5° resolution. The detailed description of the algorithm used to generate the monthly mean HIRS OLR is given in the Climate Algorithm Theoretical Basis Document (C-ATBD) for Monthly OLR CDR v02r07 [D2]. A Product User Guide and Specification document [D1] is also issued for the brokering of this CDR in the CDS.
The product has been intensively evaluated by Dr. Hai-Tien Lee (University of Maryland), the PI of the HIRS OLR product. The results are summarized in the “Quality Assurance Results and Summary” document [D5].
The developers of the CDR maintain a portal for data access and relevant documentation at:
This portal gives access to the following documents (see Table 1-1) related to the HIRS OLR CDR.
Table 1-1: Relevant documentation concerning the HIRS OLR CDR
Documents and URLs |
Schreck, C. J., H.-T. Lee and K. Knapp, 2018: HIRS Outgoing Longwave Radiation—Daily Climate Data Record: Application toward Identifying Tropical Subseasonal Variability. Remote Sens. 2018, 10, 1325. |
Lee, H.-T., 2018: Quality Assurance Summary and Results for Monthly and Daily OLR CDR (rev.20180831). |
Lee, H.-T., A. Gruber, R. G. Ellingson and I. Laszlo, 2007: Development of the HIRS Outgoing Longwave Radiation climate data set. J. Atmos. Ocean. Tech., 24, 2029–2047. |
Climate Algorithm Theoretical Basis Document (C-ATBD) for Monthly OLR CDR v02r07 |
Lee, H.-T., 2014: Daily OLR CDR – Development and Evaluation. CERES Science Team Meeting, Apr 2014 |
Lee, H.-T., 2014: Daily OLR Climate Data Record. EGU General Assembly, Apr 2014 |
Lee, H.-T., C. J. Schreck, and K. R. Knapp, 2014: Generation of Daily OLR CDR. Eumetsat Meteorological Satellite Conference, Sep 2014 |
Read me for Daily OLR CDR v01r02 |
2. Description of validating datasets
The best reference dataset for the validation of the monthly mean HIRS OLR CDR are the CERES datasets (CERES EBAF (Loeb et al., 2018) and CERES SYN1deg-month (Doelling et al., 2013 and 2016). However, the validation is then restricted to the 2000-onward period (with slightly lower quality before the inclusion of CERES Aqua in 2002).
In [D3], an intercomparison is performed between HIRS OLR, CERES EBAF and the CM SAF GERB/SEVIRI data records (Clerbaux et al., 2017). Assuming that the errors affecting those three records are uncorrelated, it is possible to estimate the individual error for the three CDRs.
3. Description of product validation methodology
The comparison with CERES EBAF is described in section 7.1 of the “Quality Assurance Results and Summary” [D5]. This provides bias and standard deviation. Each of the statistics is reported “regionally” (i.e. in 2.5°x2.5° boxes) and as a time series of global values.
The main findings of the HIRS/CERES intercomparison [D5] are reproduced in section 4.1.
In [D3], intercomparisons are performed between HIRS OLR, CERES EBAF (Edition 2.8) and the CM SAF GERB/SEVIRI ed02 data records. When at least three data records are available, it is possible to convert the observed standard deviations between the records into individual accuracies. This is only possible if the errors are not correlated. The assumption about un-correlation of the errors is supported by the fact that the data records are derived from totally different space instruments (respectively GERB/SEVIRI, CERES, HIRS) operated from different satellites/orbits (geostationary for MSG, polar for Aqua+Terra, polar for NOAA+MetOp).
By denoting the 3 data records A, B, C, and if the errors have normal distributions and are uncorrelated, we can write
\[ \sigma^2 (A-B) = \sigma^2(A) + \sigma^2(B) \] \[ \sigma^2 (A-C) = \sigma^2(A) + \sigma^2(C) \] \[ \sigma^2 (B-C) = \sigma^2(B) + \sigma^2(C) \]where s( ) is the standard deviation of either the difference (left terms) or of the error of the dataset. The previous relations can be inverted into:
\[ \sigma^2 (A) = 0.5 * (\sigma^2(A-B) + \sigma^2(A-C) - \sigma^2(B-C)) \] \[ \sigma^2 (B) = 0.5 * (\sigma^2(A-B) + \sigma^2(B-C) - \sigma^2(A-C)) \] \[ \sigma^2 (C) = 0.5 * (\sigma^2(A-B) + \sigma^2(B-C) - \sigma^2(A-B)) \]The main results are reproduced hereafter, in section 4.2.
4. Summary of validation results
4.1 Comparison with CERES EBAF
This section summarizes the main results of the intercomparison of HIRS OLR CDR v02r07 with CERES EBAF Edition 4 as reported in [D5]. The comparison is done over the period from March 2000 to February 2018.
Figure 4-1 shows the spatial variations of the mean difference between the products. In general, the HIRS OLR is slightly lower than CERES EBAF. Figure 4-2 shows the time series of the global mean difference. Although there is a clear seasonal cycle, a clear average bias exists of about 1.6 W/m² [D5]. This bias seems to remain stable during the 2002-2018 time period, probably complying with the 0.3W/m²/decade stability requirement defined by GCOS.
Figure 4-3 shows the spatial variations of the standard deviation (RMS of the difference corrected by the bias) between the products. Figure 4-4 shows the time series of the standard deviation which has an average value of 1.8 W/m² [D5]. The standard deviation remains stable during the 2002-2018 time period but shows slightly higher values before July 2002. This is likely due to degraded CERES products before July 2002, when they can use only CERES data from the Terra satellite.
Figure 4-1: Mean difference between monthly mean HIRS OLR v02r07 and CERES EBAF Edition 4, over March 2000 - February 2018. Extracted from [D5].
Figure 4-2: Timeseries of global mean difference between monthly mean HIRS OLR v02r07 (red curve) and CERES EBAF Edition 4 over Mar 2000 - Feb 2018. Extracted from [D5]. The blue curve is for an outdated version of the HIRS OLR CDR.
Figure 4-3: Standard deviation (aka “bias-corrected RMS”) of the difference between monthly mean HIRS OLR v02r07 and CERES EBAF Edition 4 over Mar 2000 - Feb 2018. Extracted from [D5].
Figure 4-4: Timeseries of the standard deviation of the difference between monthly mean HIRS OLR v02r07 (red curve) and CERES EBAF Edition 4 over Mar 2000 - Feb 2018. Extracted from [D5]. The blue curve is for an outdated version of the HIRS OLR CDR. The spikes in the plot are related to the incomplete sampling (missing days) in the CERES observations for monthly mean derivation
4.2 Comparison of "triplets" HIRS OLR / CERES SYN / CM SAF GERB
We have applied the triple collocation method to the all sky monthly mean OLR data from CM SAF, CERES SYN and HIRS. The assumption about no existing correlation between the errors is supported by the fact that the data records are derived from different space instruments (respectively GERB/SEVIRI, CERES, HIRS) operated from different satellites/orbits (geostationary for MSG, polar for Aqua+Terra, polar for NOAA+MetOp).
The result is shown in Figure 4-5. The estimated accuracies are on average: 1.6 W/m² for CM SAF and 0.9 W/m² for SYN MM (Monthly Mean) and HIRS MM. Note that those numbers are estimated over the 60°N-60°S and 60°W-60°E region and cannot be directly compared with the comparison given in section 4.1.
Figure 4-5: Individual accuracies of the monthly mean OLR (here called TOA Emitted Thermal TET) from CM SAF, CERES SYN and HIRS, as estimated from their RMS differences.
4.3 Comparison with reanalysis
The representation of radiative fluxes has significantly improved in recent reanalysis like ERA5. Although there still exist important differences with observed TOA radiation, reanalysis is however helpful to evaluate stability and detect possible jumps and artifacts in the time series.
4.4 Application(s) specific assessments
In addition to the extensive product validation (see chapter 3 for results and chapter 2/3 in [D4] for validation methodology) a second assessment is introduced to evaluate the Interim Climate Data Record (ICDR) against the Thematic Climate Data Record (TCDR) in terms of consistency. Since frequent ICDR deliveries make detailed validation not feasible, a consistency check against the deeply validated TCDR is used as an indication of quality. This is done by a comparison of the following two evaluations:
- TCDR against a stable, long-term and independent reference dataset
- ICDR against the same stable, long-term and independent reference dataset
The evaluation method is generated to detect differences in the ICDR performance in a quantitative, binary way with so called Key Performance Indicators. The general method is outlined in [D4] chapter 3. The same difference between TCDR/ICDR and the reference dataset would lead to the conclusion that TCDR and ICDR have the same quality (key performance is "good"). Variations or trends in the differences (TCDR/ICDR against reference) would require a further investigation to analyze the reasons. The key performance would be marked as "bad". The binary decision whether the key performance is good or bad is made in a statistical way by a hypotheses test (binomial test). Based on the TCDR/reference comparison (global means, monthly or daily means) a range is defined with 95% of the differences are within. This range (2.5 and 97.5 percentile) is used for the ICDR/reference comparison to check whether the values are in or out of the range. The results could be the following:
- All or a sufficient high number of ICDR/reference differences lies within the range defined by the TCDR/reference comparison: Key performance of the ICDR is "good"
- A smaller number of ICDR/reference differences is within the pre-defined range: Key performance of the ICDR is "bad"
4.4.1 Results
The results of the KPI test are summarized in Table 4-1.
Table 4-1: Summary of KPI results with 2.5 and 97.5 percentiles and number of ICDR months within the range. Colors green or red mark the results of the binomial tests as good or bad, respectively.
| Outgoing Longwave Radiation | |
|---|---|
| Percentiles | p2.5 = -5.35 W/m² p97.5 = -3.02 W/m² |
| 01/2021 - 11/2021 | 9/9 |
| 12/2021 - 04/2022 | 5/5 |
| 05/2022 - 09/2022 | Reference data not available |
| 10/2022 - 05/2023 | 8/8 |
Percentiles were calculated based on the comparison of the TCDR using the High Resolution Infrared Radiation Sounder (HIRS) instrument onboard NOAA and Metop satellites against ERA5 data for the variable Outgoing Longwave Radiation (OLR). Reference period is defined from 01/1979 to 12/2020. The complete ICDR remains within the limits leading to "good" KPI tests. The ICDR is therefore stable in relation to the deeply validated TCDR.
4.5 Discussion
Using CERES EBAF as the reference, the accuracy of the HIRS OLR product is evaluated at 1.8 W/m² [D5]. A part of this value could however be attributed to errors in the CERES products, so the uncertainty in the monthly mean HIRS OLR values is possibly better than 1.8 W/m². Assuming a similar uncertainty for HIRS and CERES products, the observed 1.8 W/m² standard deviation would reduce to 1.3 W/m² uncertainties for both products.
References
Clerbaux, Nicolas; Urbain, Manon; Ipe, Alessandro; Baudrez, Edward; Velazquez-Blazquez, Almudena; Akkermans, Tom; Hollmann, Rainer; Fuchs, Petra; Selbach, Nathalie; Werscheck, Martin, 2017: CM SAF TOA Radiation GERB/SEVIRI Data Record - Edition 2, Satellite Application Facility on Climate Monitoring, DOI:10.5676/EUM_SAF_CM/TOA_GERB/V002.
Doelling, D. R., M. Sun, L. T. Nguyen, M. L. Nordeen, C. O. Haney, D. F. Keyes, P. E. Mlynczak, 2016: Advances in Geostationary-Derived Longwave Fluxes for the CERES Synoptic (SYN1deg) Product, Journal of Atmospheric and Oceanic Technology, 33(3), 503-521. doi: 10.1175/JTECH-D-15-0147.1
Doelling, D. R., N. G. Loeb, D. F. Keyes, M. L. Nordeen, D. Morstad, C. Nguyen, B. A. Wielicki, D. F. Young, M. Sun, 2013: Geostationary Enhanced Temporal Interpolation for CERES Flux Products, Journal of Atmospheric and Oceanic Technology, 30(6), 1072-1090. doi: 10.1175/JTECH-D-12-00136.1.
Knapp, K. R., S. Ansari, C. L. Bain, M. A. Bourassa, M. J. Dickinson, C. Funk, C. N. Helms, C. C. Hennon, C. D. Holmes, G. J. Huffman, J. P. Kossin, H.-T. Lee, A. Loew, and G. Magnusdottir, 2011: Globally gridded satellite (GridSat) observations for climate studies. Bulletin of the American Meteorological Society, 92, 893-907. doi:10.1175/2011BAMS3039.1
Lee, H.-T.; Heidinger, A.; Gruber, A.; Ellingson, R.G. The HIRS outgoing longwave radiation product from hybrid polar and geosynchronous satellite observations. Adv. Space Res. 2004, 33, 1120–1124
Lee, H.-T., 2014: Daily OLR CDR – Development and Evaluation. CERES Science Team Meeting, Apr 2014
Lee, H.-T., 2014: Daily OLR Climate Data Record. EGU General Assembly, Apr 2014
Lee, H.-T., 2018: Quality Assurance Summary and Results for Monthly and Daily OLR CDR (rev.20180831).
Lee, H.-T., A. Gruber, R. G. Ellingson and I. Laszlo, 2007: Development of the HIRS Outgoing Longwave Radiation climate data set. J. Atmos. Ocean. Tech., 24, 2029–2047.
Lee, H.-T., C. J. Schreck, and K. R. Knapp, 2014: Generation of Daily OLR CDR. Eumetsat Meteorological Satellite Conference, Sep 2014
Loeb, N. G., D. R. Doelling, H. Wang, W. Su, C. Nguyen, J. G. Corbett, L. Liang, C. Mitrescu, F. G. Rose, and S. Kato, 2018: Clouds and the Earth's Radiant Energy System (CERES) Energy Balanced and Filled (EBAF) Top-of-Atmosphere (TOA) Edition-4.0 Data Product. J. Climate, 31, 895-918, doi: 10.1175/JCLI-D-17-0208.1
Loeb, N. G., S. Kato, and B. A. Wielicki, 2002: Defining top-of-atmosphere flux reference level for Earth Radiation Budget studies. J. Climate, 15, 3301-3309.
Schreck, C. J., H.-T. Lee and K. Knapp, 2018: HIRS Outgoing Longwave Radiation—Daily Climate Data Record: Application toward Identifying Tropical Subseasonal Variability. Remote Sens. 2018, 10, 1325
