Hello,

There seems to have been some change in how ERA5T data requests are processed. In short, CDS requests that used to run smoothly now fail with "UserError: Restricted access to ERA5T. Please, check that that your date selection is valid." Has this change been implemented accidentally or is this a restriction that one has to work with going forward?


In detail: Since I am not aware of an API that gives information regarding available timestamps (let alone which timestamps are ERA5 and ERA5T), I used to periodically query a time series at a single pixel for e.g. the last few days. It's meant to be a lightweight process that finds out for which timestamps data exists. E.g.:

{ "area": "0.1/-0.1/-0.1/0.1", "day": [ "24", "25", "26", "27", "28", "29", "30", "31" ], "format": "grib", "month": [ "05" ], "product_type": "reanalysis", "time": [ "00:00", "01:00", "02:00", "03:00", "04:00", "05:00", "06:00", "07:00", "08:00", "09:00", "10:00", "11:00", "12:00", "13:00", "14:00", "15:00", "16:00", "17:00", "18:00", "19:00", "20:00", "21:00", "22:00", "23:00" ], "variable": "2m_temperature", "year": [ "2022" ] }

Now (I'm writing this on 5/31/2022; around 6pm UTC), the above suddenly fails. I suspect this is due to the query looking for data e.g. on 05/31, 05/30, .... Sadly, it's not even clear what valid timestamps are. UTC "now" - 5 days (which is what is described here) MARS access restrictions#era5tRestrictedaccesstoERA5T would mean that only 2022/05/26 18:00 Z is available. At the same time the web interface suggests that 2022/05/26 21:00 Z is available.


So to repeat my question:

  • Is the above access restriction now permanent?
  • Will there be an API to learn what timestamps are available and whether the data is ERA5/ERA5T?


Thanks!

Johannes Schmude

  • No labels

37 Comments

  1. Hello Johannes, 
    I'm currently having the same issue. Did you manage to solve it?


  2. Me too, would like to know what's going on here.

    1. I have now created a JIRA support issue at ECMWF.

      1. Do you have the link to this ticket so that I can follow the investigation as well?
        Thanks a lot

        1. Not sure you have access: https://jira.ecmwf.int/plugins/servlet/desk/portal/1/CUS-20763

          It's in status "Waiting for Support" now.

          1. I don't have access indeed, please keep us posted on the outcome of the ticket

  3. Hi,

    there is currently an investigation going on regarding the way the embargo period is applied to the latest ERA5T data. 

    However it is unlikely that there will be updates until sometime next week.

    1. Thanks Michela,
      I'm still conducting some tests and it seems that even older data is not available anymore (even 5 days delay).

  4. Hi,

    we seem to have the same issue, is there any updates on this?


    1. Current status of my ticket is this:

      Dear Marco,

      Thank you for your enquiry to ECMWF Support.

      As my colleague Michela posted on the C3S Forum, there is currently an ongoing investigation about the way the embargo period is applied to the latest ERA5T data. 

      Unfortunately, due to the Jubilee holiday, it is not likely that we will be able to provide updates on this issue until next week.

      We apologise for the inconvenience this may cause in the meantime.

      Kind regards,

      Anabelle

      ECMWF Support
      Support Portal | C3S Forum| C3S Knowledge Base 

  5. Dear Michela,

    Thanks so much for keeping track of these issues. Is there any update on this?
    Thanks,

    Johannes

  6. As today, there still seem to be a problem and I get "UserError: Restricted access to ERA5T" with the following request in 'reanalysis-era5-single-levels' :

            'product_type':'reanalysis',

            'time'    : '12:00',

            'day'     : ['01','02','03','04','05','06','07','08','09','10','11','12','13','14','15',

                         '16','17','18','19','20','21','22','23','24','25','26','27','28','29','30','31'],

            'month'   : '06',

            'year'    : '2022',

            'nocache' : '47512',

            'variable': ['sea_ice_cover','sea_surface_temperature']


  7. We have been getting the same error when fetching wind data more recent than the 22nd of May:

    Open request form

    Request ID: fe6751f6-8f18-4d2c-9493-5785f6d6ba54
    Product type:
    Reanalysis
    Variable:
    100m u-component of wind, 10m u-component of wind
    Year:
    2022
    Month:
    June
    Day:
    03
    Time:
    00:00, 01:00, 02:00, 03:00, 04:00, 05:00, 06:00, 07:00, 08:00, 09:00, 10:00, 11:00, 12:00, 13:00, 14:00, 15:00, 16:00, 17:00, 18:00, 19:00, 20:00, 21:00
    Sub-region extraction:
    North 54°, West 2°, South 51°, East 6°
    Format:
    GRIB


    Show error details

    The request you have submitted is not valid

    Reason: Mars server task finished in error; UserError: Restricted access to ERA5T. Please, check that your date selection is valid. For more information, visit https://climate.copernicus.eu/climate-reanalysis [mars]; Error code is -2; Request failed; Some errors reported (last error -2)

  8. I'm also getting this error for the request:

    { "area": [8, -75, -40, -35],
    "day": "04",
    "format": "netcdf",
    "month": "06",
    "product_type": "reanalysis",
    "time": ["00:00", "01:00", "02:00", "03:00", "04:00", "05:00", "06:00", "07:00", "08:00", "09:00", "10:00", "11:00",
      "12:00", "13:00", "14:00", "15:00", "16:00", "17:00", "18:00", "19:00", "20:00", "21:00", "22:00", "23:00"],
    "variable": [
        "10m_u_component_of_wind",
      "10m_v_component_of_wind",
      "2m_dewpoint_temperature",
      "2m_temperature",
      "maximum_2m_temperature_since_previous_post_processing",
      "minimum_2m_temperature_since_previous_post_processing",
      "potential_evaporation",
      "soil_temperature_level_1",
      "surface_solar_radiation_downwards",
      "surface_thermal_radiation_downwards",
      "total_cloud_cover",
      "total_precipitation" ],
    "year": "2022" }

    Is there any update on the issue?

  9. There could be more than one issue here. If your API script attempts to retrieve say 31/04/2022 or 31/06/2022 which does not exist, then while in the past this would return no data anyway (i.e. no data returned on 31/04 or 31/06), since some changes were implemented last week, it would appear that if the data does not exist, the retrieval fails over with the error UserError: Restricted access to ERA5T (misleading I would say). 

    Of course for those of you who are trying to retrieve whatever ERA5 data is available, this means that your API scripts no longer work at this point in time. You could try modifying your scripts to test this and share your results here.


    1. Do you know of a smart way to see what timestamps are available? Because the way that the API call is now structured, you always submit all 24 hours under "time" to download one or multiple full days, but then the request will fail when the most recent day does not have all 24 hours available.


    2. Hello Anabelle,


      Is it planned to bring back the old way that when requesting a whole month of data it would still work, but provide only the latest available data for the requested month? This was the easiest option to download all the available latest data for the current month. As an example if we want to download all the data available for june 2022 without having to specify the specific days, the following request would have worked previously :


          'reanalysis-era5-single-levels',

          {

              'product_type':'reanalysis',

              'time'    : '12:00',

              'day'     : ['01','02','03','04','05','06','07','08','09','10','11','12','13','14','15',

                           '16','17','18','19','20','21','22','23','24','25','26','27','28','29','30','31'],

              'month'   : '06',

              'year'    : '2022',

              'variable': ['sea_ice_cover','sea_surface_temperature']

          },


      , but it won't work anymore with the error message "UserError: Restricted access to ERA5T". Requesting the whole current month was the easiest way to quickly get the very latest available data.


      Thank you very much.

  10. Dear Michela et al,

    Is there any update on this issue and/or its anticipated resolution date? I'm encountering this issue with my ETL for various ERA5 datasets.

  11. Just want to highlight that the current behaviour is also problematic when querying multiple years of data ending in the current time frame. Queries for multiple years would select months which are by definition not available, resulting in the above error. By default one can now not query multiple years which includes months in the current time frame which have not yet come to pass.

    In a practical sense the new behaviour limits use to the previous year, upon which the last year's data has to be parsed separately. This is real troubling for any workflow which requires multiple years upon until fairly recent time frames.


    In short, this query will work as it will query a month (January) which is available in both years.

    {
      "area": [
        1,
        -1,
        -1,
        1
      ],
      "dataset_short_name": "reanalysis-era5-single-levels",
      "day": "01",
      "format": "netcdf",
      "month": "01",
      "product_type": "reanalysis",
      "target": "download.nc",
      "time": "00:00",
      "variable": "2m_temperature",
      "year": [
        "2021",
        "2022"
      ]
    }


    In this query I want data for the whole year of 2021 but in doing so, any data queried for 2022 will also include "unavailable" months and therefore this query fails.

    {
      "area": [
        1,
        -1,
        -1,
        1
      ],
      "dataset_short_name": "reanalysis-era5-single-levels",
      "day": "01",
      "format": "netcdf",
      "month": [
        "01",
        "02",
        "03",
        "04",
        "05",
        "06",
        "07",
        "08",
        "09",
        "10",
        "11",
        "12"
      ],
      "product_type": "reanalysis",
      "target": "download.nc",
      "time": "00:00",
      "variable": "2m_temperature",
      "year": [
        "2021",
        "2022"
      ]
    }



  12. We could all get around this new problem if there was something like a "date_begin" and "date_end" set of keywords, instead of the less specific "year", "month", "day", "time" keywords.  For example, something like "date_begin": "2001-01-01 00:00".  "date_end": "2022-05-31 23:00".  Is there any such functionality already?

  13. Hi everyone

    Just to mention I am having the same issue as the other users above (getting the 'Restricted' error, error code -2) when trying to download data for 2022. Thanks ECMWF for your support.

  14. Milana, many thanks, these are good ideas.  In the first example: Is there any limit to the number of separate dates that can be specified in one request (other than the total data size of the request)?  If not, this looks like a good approach.  The middle example would work, but sends a separate request for each date, so maybe not the best for a long date range.  In the third example: I was not aware of the "/" syntax in the date value.  I will experiment with that.  Again, thanks!

  15. Dear users,

    an issue was recently discovered with the way the 5 day embargo period was being applied to requests for ERA5 data from the CDS. This has now been resolved, however a consequence of this is that now any request which includes embargoed data will fail with an error message:

    Mars server task finished in error; UserError: Restricted access to ERA5T. Please, check that your date selection is valid. For more information, visit https://climate.copernicus.eu/climate-reanalysis [mars]; Error code is -2; Request failed; Some errors reported (last error -2)

    In the past, such request would have completed without the error message. Consequently, users will have to amend their requests to exclude any embargoed dates (i.e. any dates less than 5 days before the current date).
    This change has been implemented in order to ensure access is as it should be, and we apologise for any inconvenience caused.

    Regards

    ECMWF Support

  16. Dear Michela,

    According to my tests the issue still persists, without ECMWF Support offering a solution. In short, as outlined above, if people use the CDS dataset browser to format their queries for a date range which includes multiple years (including a recent month) downloads will still fail by default. No provisions are made in the user interface, or warnings provided.

    More so, the solutions which were proposed by users and DO WORK using the date: "YYYY-MM-DD/YYYY-MM-DD" syntax were forcibly removed! I consider this a huge disservice to those looking for a solution. Basically, for multi-year queries including the current year (a common instance I would think) things are broken STILL.

    Please provide an actionable solutions,

    Best,

    Koen

  17. Is there any update on this issue?

  18. Hi All,

    I discovered a CDS API feature that allows the user to make requests over multiple months/years without triggering this error. It's possible to specify a specific start and end date for the request, and so long as the end date is more than 5 days in the past, the request should work. Here is some example syntax using the Python cdsapi:


    import cdsapi
    cds = cdsapi.Client()
    cds.retrieve(
        'reanalysis-era5-single-levels', {
            "area": [
                40.75,
                -74.25,
                40.5,
                -74
            ],
            "variable": "surface_runoff",
            "product_type": "reanalysis",
            "date": "2021-06-24/2022-06-24",
            'time': [
                '00:00', '01:00', '02:00',
                '03:00', '04:00', '05:00',
                '06:00', '07:00', '08:00',
                '09:00', '10:00', '11:00',
                '12:00', '13:00', '14:00',
                '15:00', '16:00', '17:00',
                '18:00', '19:00', '20:00',
                '21:00', '22:00', '23:00',
            ],
            "format": "grib"
        }, 'download.grib')
    
    

    This request will retrieve the ERA5 surface runoff data from the previous year for a small area as a GRIB. Hope this helps!

    1. Hi Evan,

      just to say that the recommended way to specify dates in your CDS API request (CDS ERA5 datasets) is to use the  

      'year', 'month', 'day'
      keywords in your request, as using 'date' with a date range may give unexpected results.

      Also, for this dataset, it is most efficient to request 1 month of data at a time (e.g. via a loop in the script)

      Thanks,

      Kevin

      1. Hello Kevin,


        The solutions with specifying the month and exact days need you to know in advance what are the latest dates of ERA5T that are really available. The previous behaviour in which we could request all days of the latest month and cdsapi would automatically return everything that is available was easy to get the latest dates that are really available.


        Let's say I want to download all days and hours in july 2022 without knowing in advance what are the dates that are really available, is there any ways that it can be done?


        Thank you,

        1. Hi Francois,

          No, now the only way to identify what ERA5T data are available is to look at the CDS Download data form to see the latest dates. 

          Thanks

          Kevin

          1. Kevin, under this new behavior, it would be very helpful to have a simple API call that returns the latest date/time of available data for a particular data set.  Many of us have developed code within automated processes that downloads ERA5 data, so if the only way to determine available data is to click check boxes on the menu-driven download page, none of our codes can work.  Out of desperation, people might write code that requests every hour of the last 7 days or so and see which ones fail, and thereby determine the latest data available, but I don't think that would be good for the system.  I think a simple API call to determine date of latest data is what's needed.

            1. I'd like to reiterate what Mark (and others) have said. The new system requires users to revamp their ETL codebases to either use unreliable/unpredictable workarounds (specifying by date instead of year/month/day) or overload the server with requests (Mark's out of desperation solution) for a very simple piece of information. This is ideal for nobody.


              Given all this, I would like to humbly petition that you all implement a simple API call returning the latest final and provisional data availability. I appreciate this implies real work on ECMWF's side.

              1. Hi Mark, Robert, 

                Thanks for your feedback. This feature has been recorded as a C3S 'user requirement' and so will help determine the way the CDS developed in the future,

                Hope that helps,

                Kevin

  19. Hi all,

    Is there a way to download all the latest available data automatically? I think now it is difficult because the last available date/hour is unknown (for operational purposes with the API). I'm using R.

    Thank you in advance!
    Diego

  20. Hi all,

    The latest ERA5T available data can be calculated as below:

    • Take your local time, say CEST 23:09 on 18/8/22
    • Work out UTC hours, 21:09 on 18/8/22
    • Use the UTC time and date and minus 5 days, in this case, 21:09 on 13/8/22
    • ERA5T data is available up to 21 UTC 13/8/22

    To tell the difference between ERA5 and ERA5T, you can always download data in GRIB and check the value of 'expver'.

    Hope this helps.

    Xiaobo


  21. Hello all,

    Are there any plans to add a check if data is ERA5 or ERA5T via the API? I realize you can check by downloading a file and reading the GRIB headers, however, knowing before downloading would be very helpful and save a lot of time. Especially when downloading large files. 


    Thanks,

    Travis

    1. Hi Travis,

      So far, you can download a small data in GRIB and check 'expver' in it. '1' means ERA5 and '5' means ERA5T.

      ERA5 is normally around 3-month behind the real-time.

      Hope this helps.

      Xiaobo