Post-processing of model outputs

  • mslp_${day}.grib for mean sea level pressure,
  • t2_${day}.grib for 2-meter temperature,
  • p_${day}.grib for precipitation,
  • gust_${day}.grib for 10-meter wind gust,
  • t850_${day}.grib for temperature at 850 hPa,
  • q700_${day}.grib for relative humidity at 700 hPa,
  • z500_${day}.grib for geopotential at 500 hPa,
  • u250_${day}.grib for horizontal wind components at 250 hPa,
  • u100_${day}.grib for horizontal wind components at 100 hPa,

where day is the given date in format of yyyymmdd (e.g., 20151203). All the files should contain data for 8 timesteps per day (i.e., in every 3 hours).

Please note the lowercase letters in the filenames.

The mean sea level pressure, the 2-meter temperature, the precipitation and the wind gust are expected in gridpoint representation. So they are prepared from the raw ICMGG* outputs with the next operations:









<ul style="margin-left: -20px;">
<li><a href="https://confluence.ecmwf.int/display/ECC/ecCodes+Home" target="_blank">ecCodes</a>: installation, documentation</li>
<li>Access to ECMWF public datasets via <a href="https://confluence.ecmwf.int/display/WEBAPI/Access+ECMWF+Public+Datasets" target="_blank">WebAPI</a>: installation, documentation</li>
<li><a href="https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim" target="_blank">ERA-Interim data documentation</a></li>
<li><a href="https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5" target="_blank">ERA5 data documentation</a></li>
</ul>




% grib_copy -w shortName=msl  ICMGG${expID}+00${step} mslp_${day}_${step}.grib  #to get the mean sea level pressure
% grib_copy -w shortName=2t   ICMGG${expID}+00${step} t2_${day}_${step}.grib    #to get the 2-meter temperature
% grib_copy -w shortName=10fg ICMGG${expID}+00${step} gust_${day}_${step}.grib  #to get the 10-meter wind gust

where expID is the 4-digit experiment ID and step is the post-processing step (corresponding to every 3 hours) in 4 digits. For precipitation, both the convective and large-scale precipitation components have to be gathered in the same file:

% grib_copy -w shortName=lsp/cp ICMGG${expID}+00${step} p_${day}_${step}.grib

The pressure level data are required in spectral representation. So they are prepared from the raw ICMSH* outputs with the next operations:

% grib_copy -w shortName=t,level=850 ICMSH${expID}+00${step} t850_${day}_${step}.grib  #to get the temperature at 850 hPa
% grib_copy -w shortName=r,level=700 ICMSH${expID}+00${step} q700_${day}_${step}.grib  #to get the relative humidity at 700 hPa
% grib_copy -w shortName=z
,level=500 ICMSH${expID}+00${step} z500_${day}_${step}.grib  #to get the geopotential at 500 hPa

For wind, both the u and v components have to be collected in the same file:

% grib_copy -w shortName=u/v,level=250 ICMSH${expID}+00${step} u250_${day}_${step}.grib #to get the u and v components at 250 hPa
% grib_copy -w shortName=u/v,level=100 ICMSH${expID}+00${step} u100_${day}_${step}.grib #to get the u and v components at 100 hPa

After these operations, the timesteps belonging to the same days have to be merged into a common file:

% cat ${variable}_${day}_*.grib > ${variable}_${day}.grib

The size of the resulted files varies by the spatial resolution and the representation of the data. For instance, the file size at T255L91 resolution is 10 MB and 8 MB per variables for gridpoint and spectral fields, respectively, whereas these values increases to 35 MB and 26 MB at T639L137, to 233 MB and 179 MB at T1279L137.

Preparation of reference data

  • ${dataset}_mslp_${period}.grib for mean sea level pressure,
  • ${dataset}_t2_${period}.grib for 2-meter temperature,
  • ${dataset}_p_${period}.grib for total precipitation,
  • ${dataset}_gust_${period}.grib for 10-meter wind gust,
  • ${dataset}_t850_${period}.grib for temperature at 850 hPa,
  • ${dataset}_q700_${period}.grib for relative humidity at 700 hPa,
  • ${dataset}_z500_${period}.grib for geopotential at 500 hPa,
  • ${dataset}_u250_${period}.grib for horizontal wind components at 250 hPa,
  • ${dataset}_u100_${period}.grib for horizontal wind components at 100 hPa,

where dataset is a 2-digit identifier of the re-analysis data, ei for ERA-Interim and ea for ERA5; period is the investigated time period in format of yyyymmdd-yyyymmdd (e.g., 20151201-20151206 for Desmond).

Please note again the lowercase letters in the filenames. Furthermore, the re-analysis data should not be split by day, because data for the whole period will be handled together by the Metview macros.

The mean sea level pressure, the 2-meter temperature, the precipitation and the wind gust are expected in gridpoint representation, while the pressure level data are required in spectral representation. Total precipitation and wind gust as parameters representing a time period derive from forecasts, all the other variables are real analyses. Consequently, the daily quantities for precipitation and wind gust are composed of 8 and 24 timesteps from ERA-Interim and ERA5, respectively, the other variables have 4 and 8 timesteps (recall that output frequency of the forecast experiment is 3 hours). Besides the two (large-scale and convective) precipitation components, total precipitation is also available for direct retrieve both in ERA-Interim and ERA5, with GRIB code 228.

The necessary re-analysis data are available from the ECMWF download server in the proper format. For more information please visit http://download.ecmwf.int/test-data/openifs/reference_casestudies.

The re-analysis data can be retrieved directly from the MARS archive, too.



<p>
The <span class="font_code_black">scr_download_re-analysis</span> shell script is prepared <b>to get the necessary data from MARS</b>. The following steps have to be accomplished:
<ol>
   <li class="li_withspace">Using the <span class="font_code_black">scr_download_re-analysis</span> script requires <b>Python 2.7 or higher version</b>. Please make sure that the right Python version is installed on your computer.
   <li class="li_withspace">ERA-Interim data are available in the ECMWF public datasets. To have access to the ECMWF public datasets, an account is needed to the ECMWF web site: <a href="https://apps.ecmwf.int/registration/" target="_blank">https://apps.ecmwf.int/registration/</a>.
   <li class="li_withspace">To retrieve data from MARS through ecmwfAPI, an ECMWF key is needed to be downloaded and installed. This page shows that step by step: <a href="https://confluence.ecmwf.int/display/WEBAPI/Access+ECMWF+Public+Datasets" target="_blank">Access ECMWF Public Datasets</a>.
   <li class="li_withspace">Having the key, the <a href="http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars/scr_download_re-analysis" target="_blank"><span class="font_code">scr_download_re-analysis</span></a> shell script has to be run. It is available in the <span class="font_code_black">casestudies/mars</span> directory in the OpenIFS cycle 40r1v2 release as well as in the ECMWF download server: <a href="http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars/" target="_blank">http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars</a>. It uses <a href="http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars/get_ecmars" target="_blank"><span class="font_code">get_ecmars</span></a> for retrieve which is also located both in the OpenIFS cycle 40r1v2 release (in the <span class="font_code_black">bin</span> directory) and in the ECMWF download server.
   <li class="li_withspace">After submission of the retrieval, it can be checked in the <a href="http://apps.ecmwf.int/webapi-activity" target="_blank">Web-API job list</a> (after login). There is also possibility here to cancel unwanted jobs.
</ol>
</p>

The re-analysis source (i.e., ERA-Interim or ERA5), the surface and pressure level variables to be retrieved, the period of the data and the output directory have to be specified as options for the scr_download_re-analysis script. Calling it with -h option, it provides a detailed help with some examples at the end (but calling it without any option, it also gives a short instruction to its configuration):

% ./scr_download_re-analysis -h

 ----------------------------------------------------------------------
 This script downloads surface and pressure level ERA-Interim or ERA5 re-analysis data for a given time range from MARS.

 Usage:
        -c[class] -s[surface_variables] -p[plevel_variables] -f[firstdate:yyyymmdd] -l[lastdate:yyyymmdd] -o[output_directory]
        -h/-help

 Examples:
        ./scr_download_re-analysis -cei -s t2,p,mslp,gust -p t850,q700,z500,u250,u100 -f20151203 -l20151205 -o../reference
        ./scr_download_re-analysis -cea -s all -p t850,q700,z500,u250,u100 -f20151203 -l20151205 -o/home/rd/digs/metview/paper_OIFS/input/reference
        ./scr_download_re-analysis -ce5 -p t850 -f20151203 -o/home/rd/digs/metview/paper_OIFS/input/reference
  ----------------------------------------------------------------------


Please take into account that the script is able to retrieve only the variables listed above. For further parameters, the program has to be modified manually.

Furthermore, please note the all option which can be used with -s and -p switches. That takes automatically all the surface and/or pressure level variables discussed above (instead of listing them in the command line).




<p>
The <span class="font_code_black">scr_download_re-analysis</span> shell script is prepared <b>to get the necessary data from MARS</b>. The following steps have to be accomplished:
<ol>
   <li class="li_withspace">Using the <span class="font_code_black">scr_download_re-analysis</span> script requires <b>Python 2.7 or higher version</b>. Please make sure that the right Python version is installed on your computer.
   <li class="li_withspace">Since March 2019, ERA5 data are available through Copernicus Climate Data Store (CDS). To have access to the data, an account is needed to CDS: <a href="https://cds.climate.copernicus.eu/user/register" target="_blank">https://cds.climate.copernicus.eu/user/register</a>.
   <li class="li_withspace">You will need the <b>CDS API</b> client. It is a python based library, providing support for both Python 2.7.x and Python 3. You can install the CDS API client via the package management system pip, by running on Unix/Linux this command: <span class="font_code_black font_background_grey">% pip install cdsapi</span>.
   <li class="li_withspace">To retrieve data through CDS API, a CDS key is needed to be downloaded and installed. This page shows that: <a href="https://cds.climate.copernicus.eu/api-how-to" target="_blank">How to use the CDS API</a>.
   <li class="li_withspace">Before you may download any data from the CDS, you need to have accepted the <i>Terms and Conditions</i> of the chosen dataset. To do this:
   <ol>
      <li>Go to the <a href="https://cds.climate.copernicus.eu/#!/home" target="_blank">C3S Climate Data Store</a>,
      <li>On the top menu bar, click on <span class="font_code_black">Datasets</span>,
      <li>On the left-hand side menu, expand <span class="font_code_black">Product type</span> and select Product type of interest (e.g. <span class="font_code_black">Reanalysis</span> for ERA5 datasets),
      <li>Follow the dataset title link of interest to the full dataset record,
      <li class="li_withoutspace">Accept dataset licence.
   </ol>
   <li>Having the key, the <a href="http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars/scr_download_re-analysis" target="_blank"><span class="font_code">scr_download_re-analysis</span></a> shell script has to be run. It is available in the <span class="font_code_black">casestudies/mars</span> directory in the OpenIFS cycle 40r1v2 release as well as in the ECMWF download server: <a href="http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars/" target="_blank">http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars</a>. It uses <a href="http://download.ecmwf.int/test-data/openifs/reference_casestudies/programs/mars/get_ecmars" target="_blank"><span class="font_code">get_ecmars</span></a> for retrieve which is also located both in the OpenIFS cycle 40r1v2 release (in the <span class="font_code_black">bin</span> directory) and in the ECMWF download server.   
</ol>
</p>

The re-analysis source (i.e., ERA-Interim or ERA5), the surface and pressure level variables to be retrieved, the period of the data and the output directory have to be specified as options for the scr_download_re-analysis script. Calling it with -h option, it provides a detailed help with some examples at the end (but calling it without any option, it also gives a short instruction to its configuration):

% ./scr_download_re-analysis -h

 ----------------------------------------------------------------------
 This script downloads surface and pressure level ERA-Interim or ERA5 re-analysis data for a given time range from MARS.

 Usage:
        -c[class] -s[surface_variables] -p[plevel_variables] -f[firstdate:yyyymmdd] -l[lastdate:yyyymmdd] -o[output_directory]
        -h/-help

 Examples:
        ./scr_download_re-analysis -cei -s t2,p,mslp,gust -p t850,q700,z500,u250,u100 -f20151203 -l20151205 -o../reference
        ./scr_download_re-analysis -cea -s all -p t850,q700,z500,u250,u100 -f20151203 -l20151205 -o/home/rd/digs/metview/paper_OIFS/input/reference
        ./scr_download_re-analysis -ce5 -p t850 -f20151203 -o/home/rd/digs/metview/paper_OIFS/input/reference
  ----------------------------------------------------------------------


If you have other possibility to retrieve the ERA5 re-analysis data (e.g. via MARS client), please consider to use those one in order to avoid the sometimes long-lasting queues in CDS.

Please take into account that the script is able to retrieve only the variables listed above. For further parameters, the program has to be modified manually.

Furthermore, please note the all option which can be used with -s and -p switches. That takes automatically all the surface and/or pressure level variables discussed above (instead of listing them in the command line).


The retrieved ERA-Interim fields occupy approximately 54 MB, while the ERA5 fields take 700 MB for the Desmond case (i.e., for 1–6 December 2015).

References

<p><a name="dee"></a>Dee, D.P., Uppala, S.M., Simmons, A.J., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M.A., Balsamo, G., Bauer, P., Bechtold, P., Beljaars, A.C., van de Berg, L., Bidlot, J., Bormann, N., Delsol, C., Dragani, R., Fuentes, M., Geer, A. J., Haimberger, L., Healy, S.B., Hersbach, H., Hólm, E.V., Isaksen, L., Kållberg, P., Köhler, M., Matricardi, M., McNally, A.P., Monge‐Sanz, B.M., Morcrette, J., Park, B., Peubey, C., de Rosnay, P., Tavolato, C., Thépaut, J., Vitart, F., 2011: The ERA‐Interim reanalysis:
configuration and performance of the data assimilation system. <em>Q.J.R. Meteorol. Soc. 137,</em> 553&ndash;597. doi: <a href="https://doi.org/10.1002/qj.828" target="_blank">10.1002/qj.828</a></p>
<p><a name="hersbach_2016"></a>Hersbach, H., Dee, D.P., 2016: <a href="https://www.ecmwf.int/en/newsletter/147/news/era5-reanalysis-production" target="_blank">ERA5 reanalysis is in production.</a> <em>ECMWF Newsletter 147,</em> p. 7.</p>
<p><a name="hersbach_2018"></a>Hersbach, H., de Rosnay, P., Bell, B., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Alonso-Balmaseda, A., Balsamo, G., Bechtold, P., Berrisford, P., Bidlot, J-R., de Boisséson, E., Bonavita, M., Browne, P., Buizza, R., Dahlgren, P., Dee, D., Dragani, R., Diamantakis, M., Flemming, J., Forbes, R., Geer, A., Haiden, T., Hólm, E., Haimberger, L., Hogan, R., Horányi, A., Janisková, M., Laloyaux, P., Lopez, P., Muñoz-Sabater, J., Peubey, C., Radu, R., Richardson, D., Thépaut, J-N., Vitart, F., Yang, X., Zsótér, E., Zuo, H., 2018: <a href="https://www.ecmwf.int/sites/default/files/elibrary/2018/18765-operational-global-reanalysis-progress-future-directions-and-synergies-nwp.pdf" target="_blank">Operational global reanalysis: progress, future directions and synergies with NWP.</a> <em>ECMWF ERA Report Series 27</em>.


Back to top


.cell_nopadding {
    padding: 3px;
    border: 0px;
}

.cell_leftmargin {
    padding-left: 30px;
}

.cell_top-bottom-border {
    border-top: 1px solid #3572b0;
    border-bottom: 1px solid #3572b0;
    border-left: 0px;
    border-right: 0px;
    padding: 3px;
}

 .table_noborder {
    border: 0px;
}

.vertical {
    height: 100%;
}

.vertical td:first-child {
    position: relative;
    width: 35px;
    padding: 0;
    border: 2px solid #ddd;
}

.vertical_text {
    display: block;
    white-space: nowrap;
    -moz-transform: rotate(-90deg);
    -moz-transform-origin: center center;
    -webkit-transform: rotate(-90deg);
    -webkit-transform-origin: center center;
    -ms-transform: rotate(-90deg);
    -ms-transform-origin: center center;
    position: absolute;
    width:25px;
    color: #205081;
}

.table_line {
    padding: 0;
    height: 2px;
    border-left: 0px;
    border-right: 0px;
}

.table_line td:first-child {
    padding: 0;
    height: 2px;
    border-left: 0px;
    border-right: 0px;
}

.font_background_cyan {
    background-color: cyan;
}

.font_background_yellow {
    padding: 2px;
    background-color: #ffffb3;
}

.font_background_grey {
    padding: 2px;
    background-color: #dddddd;
}

.font_code_black {
    color: #000000;
    font-family: Monospace;
}

.font_code {
    font-family: Monospace;
}

.li_withspace {
    margin-bottom: 10px;
}
.li_withoutspace {
    margin-bottom: -20px;
}
.img_withoutspace {
    margin-bottom: -12px;
}