Skip to end of metadata
Go to start of metadata

Introduction

Main steps

Main steps:

  • We strongly recommend users to read our ECMWF Service Status and the Web-API FAQ before reporting the problem
  • If the problem you are experiencing is included in the troubleshooting list below, please follow the corresponding instructions.
  • If you are still experiencing problems, or if the problem is NOT included in the troubleshooting list below, please feel free to report the problem

(lightbulb) Before you report any problem please note that usually on Wednesdays, we carry out service maintenance sessions, which may affect the availability of the Web-API service.

  • For real time information please visit the ECMWF Service Status area.     
  • Please note that Web API availability, depends on the availability of other services like MARS, INTERNET and WEB-SERVICES.


How to report a problem

How to report a problem:

Who to contact: servicedesk@ecmwf.int

What information you will need to pass:

  • Your login
  • Your job id 
  • The problem you are experiencing

Where to find this information:

  • All this information can be found on your job list (see next paragraph) or
  • In the service response (in case of a Web-API  script, look the log messages returned by the server)
  • A lot of useful information can be also found on the MARS activity page

Your job list

Your job list

(lightbulb) Your job list is important for tracing back and debugging your requests. It includes plenty of useful information, like the following:

  • job id,
  • user login,
  • status of your request,
  • duration of your request,
  • details of your request (ie what you have requested),
  • log messages of your request (ie the feedback from our servers).  
  • an option to cancel a request (sad)

 

killing or cancelling a request?

(warning) If you kill a Web-API request on your local environment (e.g. by CTRL+C), the corresponding job, on the web-API service level, is NOT cancelled but still running. In that case please keep in mind the following:

  • Your request will continue to be visible in your job list
  • If you wish to cancel it please use the corresponding option on your job list
  • Once you have cancelled it, the request status will become aborted.
  • Due to the "max active requests per user" limitation, killing or cancelling requests may affect the performance of your other submitted requests.


1. General issues

Browse our online Web-API FAQ for help with common user questions.

1.1 long running request

  • Check the status of your request in your job list
  • If your request is not in your job list please report the problem
  • If your request is in your job list and the status has been "queued", for a long time, we recommend you the following:
    • Check the Nr of your "active" and "queued" requests. Please keep in mind that a limitation of  3  "active" requests/per user has been set to improve the service
    • Have a look on the Web-API activity to check the current status of our service.(ie if the service is busy you will see plenty of "queued"  requests). You may also wish to check our  MARS activity
  • If your long running request is in your job list and the  status  is "active"it means that your request is currently handled by MARS but for some reasons it is slow.
    In this case we recommend you the following:

 

(lightbulb) Please keep in mind the following:

  • Many requests submitted by other users may be running in parallel consuming the services resources (MARS slots, Web-API slots, Tapes etc)
  • A limitation of  3 "active" requests  and 20 "queued" requests per user has been set to improve the service
  • Read carefully our retrieval efficiency pages and check if you can split your big requests
  • Read the efficiency tips below:

Efficiency tips

  • Smaller Nr of tapes (a request accesses) means better request efficiency.
  • Ideally requests should not access more than 1 tape.
  • Requests for data which is online in general go faster
  • All this information above, can be found in MARS activity

1.2 aborted request

  • If your request is "aborted" please check the log messages, find the "error"  and follow the troubleshooting instructions below.
  • If the problem is not included  in the troubleshooting list below, please contact us.

1.3 restricted access to valid data

example
'mars - ERROR  - 20160712.064620 - Request validation failed:',
'mars - ERROR  - 20160712.064620 - restricted access to valid data.',
'mars - ERROR  - 20160712.064620 - For more information, look at ',
'mars - ERROR  - 20160712.064620 - https://software.ecmwf.int/wiki/display/UDOC/MARS+access+restrictions#valid.',
'mars - ERROR  - 20160712.064620 - For any queries, please, contact User Support',
'mars - ERROR  - 20160712.064620 - Some errors reported',
  • Your request will be  "aborted" if you are trying to access restricted data.
  • For more information please have a look at MARS access restrictions

1.4 Cannot retrieve more than XXX.XXX fields in one request via the WebAPI. Please, split your request

example
mars - ERROR  - 20160712.133205 - UserError: Cannot retrieve more 
than XXX.XXX fields in one request via the WebAPI. Please, split your 
request [marsode]
mars - ERROR  - 20160712.133205 - Error code is -2
mars - ERROR  - 20160712.133205 - Request failed
mars - INFO   - 20160712.133205 - Request time:  wall: 1 sec
mars - ERROR  - 20160712.133205 - Some errors reported
  • On Web-API some limitations have been set to improve the service.  Check the current limits here: News feed 
  • In this case, your job has reached the limit of XXX.XXX fields per request. Please split your request into smaller sub requests.

1.5 ecmwfapi.api.APIException: u'ecmwf.API error 1: TaskError: Maximum retrieve size XX.XX GB reached. Please split your request.'

  • On Web-API some limitations have been set to improve the service.  Check the current limits here: News feed 
  • In this case your job has reached the limit of XXGB per request. Please split your request into smaller sub requests.



2. HTTP status issues

2.1 HTTP Error 404: NOT FOUND

2.1.1 Resource not found: datasets/None'

if you are using the Access ECMWF Public Datasets service and you get this error it means that the mandatory keyword "dataset" is not set. Review the request and add this keyword.

If you are an authorized user with MARS access you can use WebAPI FAQ#Can I use the Python requests to retrieve data using API MARS service?

2.1.2 Resource not found: datasets/interim_daily or Resource not found: datasets/interim_full

The datasets interim_daily and interim_full do not exists any more. Please change your request to ask for the dataset interim instead. Please note that interim is now equivalent to the interim_full, the full resolution dataset. To get the equivalent to the interim_daily dataset, you will need to set "grid" to "1.5/1.5" in your request.

2.2 HTTP Error 403: FORBIDDEN

2.2.1 ecmwfapi.api.APIException: u"ecmwf.API error 1: User '...' has no access to services/mars"

2.3 HTTP 429 errors

2.3.1 HTTP 429: Too Many Requests

Reason: The rate limit of the http requests  on the API service has been exceeded.

Current limit: 10 http requests/sec

Action: Please try again after 1 minute.

WARNING: HTTP received 429
{
  "message": "API rate limit exceeded.",
  "type": "rate_limit_exceeded",
  "documentation_url": "https://software.ecmwf.int/wiki/display/WEBAPI/Web-API+Troubleshooting#Web-APITroubleshooting-HTTP429errors"
}
Error contacting the WebAPI, retrying in 60 seconds ...

2.3.2 user_queued_limit_exceeded

Reason: The Nr of queued requests per user on the API service has been exceeded

Current limit20 queued requests per user

Action: Please limit the Nr of your queued requests and try later

{
    "code": 429,
    "error": "Too many queued requests (1). Max allowed queued requests per user is 20.",
    "type": "user_queued_limit_exceeded",
    "documentation_url": "https://software.ecmwf.int/wiki/display/WEBAPI/Web-API+Troubleshooting#Web-APITroubleshooting-user_queued_limit_exceeded"
}
Error contacting the WebAPI, retrying in 60 seconds ...


2.3.3 system_queued_limit_exceeded

Reason: The total Nr of queued requests in the API service has been exceeded. Presumably the API service is currently busy. Please accept our apologies for any inconvenience this may cause.

Current limit: 2000 requests in total

Action:  try  later you may also wish to check the  Web-API activity

WARNING: HTTP received 429
{
    "code": 429,
    "error": "Too many queued requests. Global limit for queued requests hit."},
    "type": "system_queued_limit_exceeded",
    "documentation_url": "https://software.ecmwf.int/wiki/display/WEBAPI/Web-API+Troubleshooting#Web-APITroubleshooting-system_queued_limit_exceeded",
}
Error contacting the WebAPI, retrying in 60 seconds ...

2.4 HTTP Error 5xx

Please check the ECMWF Service Status before reporting the problem

If the WEB-SERVICES status is green, it might be an Internal ECMWF server issue. Please report the problem

3. Web-API issues

3.1 Invalid or missing key or token

Make sure you get a valid key from https://api.ecmwf.int/v1/key/. The file in $HOME/.ecmwfapirc should contain something like:

{
    "url"   : "https://api.ecmwf.int/v1",
    "key"   : "XXXXXXXXXXXXXXXXXXXXXX",
    "email" : "john.smith@example.com"
}

Including the curly brackets.

3.2 Python issues

3.2.1 ECMWF API at None

3.2.1.1 ValueError: unknown url type: None/who-am-i?offset=0&limit=500

 

3.2.1.2 WARNING: httplib2.URLError received None <urlopen error unknown url type: https>

Access to ECMWF servers is done over HTTPS. Your Python version does not support SSL (Secure Socket Layer) used by HTTPS.

To verify if SSL is enabled in your python , try the following:

>>> import socket
>>> socket.ssl
<function ssl at 0x4038b0>


If the output is not <function ssl at some_hex_number>, then you should rebuild or reinstall python with ssl enabled.

 

3.2.1.3 WARNING: httplib2.URLError received None <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

Symptoms: A Python WebAPI script terminates with "Error contacting the WebAPI, retrying in 60 seconds ...".  The WebAPI retries 10 times, followed by "WARNING: httplib2.URLError received None <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed"

The problem may be that the SSL library used in this Python version doesn't look in the correct certificate path.

  • Try from another environment.
  • Check which version of Python you are using and how it was installed and linked to openssl.
  • Check that ca-certificates are updated on your system. Important, the client, openssl on most systems, maintains a set of trusted root CAs that are used to validate the chain (intermediate cert > server cert) on a ssl connection, so they must be updated on the client.
    • Update OpenSSL, ca-certificates and Python.
    • The following are some references on how to update the ca-certificates package:
Mac OS X
    sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/new-root-certificate.crt
    # Do not update OpenSSL through Homebrew, see https://stackoverflow.com/questions/15185661/update-openssl-on-os-x-with-homebrew

Windows
	# Type mmc.exe on search bar of the windows menu
	# Go to File > Add / Remove Snap In > Select Certificates > Click Add > Click OK
	# Expand Certificates - Current User
	# Right Click on Trusted Root Certification Authorities > All Tasks > Import
	# Follow the steps

	# alternatively
    certutil -addstore -f "ROOT" new-root-certificate.crt

Linux (Ubuntu, Debian)
    apt install ca-certificates
    # Copy your CA to dir /usr/local/share/ca-certificates/
    sudo cp foo.crt /usr/local/share/ca-certificates/foo.crt
    sudo update-ca-certificates

Linux (CentOs 6)
    yum install ca-certificates
    #Enable the dynamic CA configuration feature:
    update-ca-trust force-enable

    #Add it as a new file to /etc/pki/ca-trust/source/anchors/:
    cp foo.crt /etc/pki/ca-trust/source/anchors/

    update-ca-trust extract

Linux (CentOs 5)
    #Append your trusted certificate to file /etc/pki/tls/certs/ca-bundle.crt
    cat foo.crt >> /etc/pki/tls/certs/ca-bundle.crt

Anaconda
	conda update openssl
    • In case you need to add a specific Root Certificate, run the following command (The browser can be used to get the same information, if not on a *nix system, on view certificate):
user@local:~> openssl s_client -connect api.ecmwf.int:443                                                                                  
CONNECTED(00000003)
depth=2 C = BM, O = QuoVadis Limited, CN = QuoVadis Root CA 2 G3
...

Here we can see QuoVadis Root CA 2 G3 is the Root CA for api.ecmwf.int and can be downloaded from https://www.quovadisglobal.com/QVRepository/DownloadRootsAndCRL.aspx and installed locally (as per the commands specified above on the specific OS).

If you have the SSL error in the last step of your request (Transferring from...), you need to install the Root CA for "stream.ecmwf.int": QuoVadis Global SSL ICA G2

  • The following is an hack for cases when updates are not possible for whatever reason:

    • Disable SSL Verification, this can be achieved by setting CURL_CA_BUNDLE="" before calling the python api:
      • CURL_CA_BUNDLE="" python main.py
    • Specify the Root CA directly, this can be achieved by setting REQUESTS_CA_BUNDLE="path to ROOT ca QuoVadis Root CA 2 G3" downloaded from the Quovadis Website (that your system cannot find somehow):
      • REQUESTS_CA_BUNDLE="/path_to_cert/QuoVadis_Root_CA_2_G3.pem" python main.py

 

3.2.2 Python socket module was not compiled with SSL support. Aborting...

Access to ECMWF servers is done over HTTPS. Your Python version does not support HTTPS. See above on how to check the ssl support in Python.

3.2.3 assert total == size AssertionError or "Transfer interrupted, retrying..."

This means that the Web API client script failed to transfer the data from the ECMWF servers to your local machine. The reasons can be:
  • Connectivity network issues.
  • Local disk space or file systems issues

(lightbulb) Check if the results are still available in your job list

(smile) If not, you may wish to rerun the request

(warning) Check if you have installed the latest api client from our Web-API Downloads

 

3.3 Perl issues

Can't connect to ... (certificate verify failed) or LWP::Protocol::https::Socket: SSL connect attempt failed with unknown error

Try to set the environment variable PERL_LWP_SSL_VERIFY_HOSTNAME to 0:

export PERL_LWP_SSL_VERIFY_HOSTNAME=0

or directly in your Perl script:

BEGIN { $ENV{"PERL_LWP_SSL_VERIFY_HOSTNAME"} = 0; }

4. ecCodes issues

grib_to_netcdf: Why do I get the error "Wrong number of fields... Try using the -T option"?

For any kind of ecCodes issues please visit the  ecCodes Frequently Asked Questions

5. MARS issues

General MARS Troubleshooting

  • If you are experiencing a MARS "error" please check also the General MARS Troubleshooting page.
  • Some additional information on some specific issues has  also been added in the list  below.

5.1 MARS messages

(lightbulb) It is very important users to be aware of the MARS messages:

INFOrequest being processed and a report on the execution at the end
DEBUGadditional information if debugging is switched on
WARNINGany unusual aspect of the execution
ERRORsystem or data errors which do not stop MARS execution
FATALterminates the execution of MARS

5.2 mars - WARN   - 20130522.082222 - connect : marsxyz 9000 (Connection refused)

5.3 mars - FATAL - 20160712.162606 - Signal 15 received

5.4 ERROR 1 MARS_REQUEST_TOO_LARGE

  • Status: aborted
  • Reason: The volume of your request is too large 
  • Options: Split your request to smaller ones

5.5 ERROR 2 MARS_TOO_MANY_FIELDS

  • Status: aborted
  • Reason: You are requesting too many fields
  • Options: Split your request to smaller ones

5.6 ERROR 3 (or 4 or 5)  MARS_RESTRICTED_ACCESS_TO_XXX

  • Status: aborted
  • Reason: You try to access restricted data
  • Options: Adapt your request

5.7 ERROR 6 MARS_EXPECTED_FIELDS

  • Status: aborted
  • Reason:  Retrieved Nr of fields != expected Nr of fields. Some of the fields that you have requested are not available. eg some S2S parameters are not available from all pressure levels.
  • Options: Check the availability of the dataset using the ECMWF Public Datasets web interface and adapt your request to reflect the availability.

5.8 ERROR 99

For any other MARS problem you can visit the MARS Trouble-shooting


  • No labels