1. Introduction


This document describes the service that allows users to automatically submit jobs to be run when certain points in the daily ECMWF operational forecast suites have been reached. The main purpose is to ensure that certain data is available before e.g. submitting a MARS request. This facility is running using the ECaccess environment. It is available either through the Web interface of ECaccess or with the ECaccess Web Toolkit, available on the Atos HPC or installed locally.  This service is monitored by the operators at ECMWF.

Before submitting your job to run in the ECaccess environment, you need to set up SSH key-based authentication within ECMWF as explained in HPC2020: How to connect.

2. Enhanced ECaccess batch service

In 2007, the existing batch service under ECaccess was extended to provide a new facility allowing registered users to run jobs when ECMWF’s operational activity has reached certain points.

2.1. Events

A database of events, also known as notifications, has been added to ECaccess. Such events can be added, deleted or modified by individual users. An event will have a name and a description. In the context of this service for registered users, the events defined in ECaccess correspond to the points in the operational suite when some data or products are available. For example, we have defined an event called ‘an12h00’ with the description ‘At this stage, the analysis cycle for 12:00UTC is complete.’ As such, this event will not yet have any link with the ECMWF operational activity, apart maybe by its name and description. From the user viewpoint, when submitting a batch job through ECaccess, the user will be able to subscribe (repeatedly) this job to the events available to him or her. And from the ECMWF operational viewpoint, we will send notifications to these predefined events. When ECaccess receives a notification for an event, it will release the user jobs which have subscribed to the event and submit them to the Slurm batch service on the system on the Atos HPCs. Finally, a little time after a notification to an event has been issued and the jobs subscribing to the event have been submitted, a ‘sweeper’ daemon within ECaccess will prepare a new version of the users’ jobs subscribing to the event, ready for submission at the next notification of the event.

2.2. User interface

The jobs to be attached to the ECMWF operational suite will have to be submitted through ECaccess. Batch job submission is available from the ECaccess web interface or through the ECaccess Web Toolkit. We will first look at the Web interface, then at the Web Toolkit.

2.2.1. Web interface

When logged in on the web interface for ECaccess, e.g. at https://boaccess.ecmwf.int/ or on your local gateway, you have the possibility to submit a new job from the left margin.

The upper part of the submission page is shown in figure 1.

Figure 1: Job submission - Upper part

You can either type in your job script in the window provided, copy and paste it from another window or upload it from a local file. One important addition to make to your jobs is to add the "set -e" command or alternatively to manage the errors in your jobs and exit accordingly - see Job status, below, for more details.

Figure 2: Job submission - lower part

The lower part of the job submission window (see Figure 2) - called subscription - allows you to attach your job to the different events available to you. Simply tick the boxes corresponding to the event(s) when you want to run your job.

By default, the jobs you attach to an event will be run automatically every time a notification is sent to the given event. If you want a job only to be run at the first next notification of an event, you can untick the box labelled ‘automatically renew subscription’. Under point ‘Settings of your job request’ you can customise various options for your job. The important options for this service are described below:

  1. Keep job input/output for:  ECaccess will create one new job for subsequent notifications of an event. E.g. if you have subscribed one job to the event ‘an00h000’ one new ECaccess job will automatically
    be created and submitted every day when the ECMWF analysis for 00UTC is complete. The jobs used for the previous days will be kept in the ECaccess spool; they will be removed after the number of days specified in this field.
  2. Man page for your job:  The ECMWF operators have utilities to monitor your jobs subscribing to any events of the ECMWF operational activity. In this page, you can give some instructions to the operators
    on what to do if the job fails. Operators could rerun your job (see next point) or possibly inform someone about the problem. If no instructions are given, our operators will not take any specific action on your jobs.
  3. Retry frequency and Retry count:  With these options, you can request your job to be rerun automatically (without the intervention of the ECMWF operators) a certain number of times if it fails.   We recommend you use submit your job with Retry count set to 3 and Retry frequency to 15 minutes (900 seconds) to allow the job to retry 3 times with 15 minutes between job resubmissions. This will sometimes allow the job to complete successfully if the initial failure was caused by a temporary issue.
  4. One script to one notification:  If you have ticked several events for your job, by default you will have one job running for each individual event. If you want only one job to run when all the notifications t the events have been received, you can untick the box labelled ‘one script to one notification’. This option could be used if, for example, you want to extract in the same job  some ENS meteogram products and raw ENS data. Your single job will seem to be linked to the two events and will be submitted when a notification to the two events has been sent. Be careful, though, to submit such a job at the correct moment, before the two events occur in the Operational suite, not in between. When you have given all the necessary information about your job, you can submit it. Your job will be taken by ECaccess and put in standby mode - status STDBY. You can monitor your jobs by selecting the link ‘Job submission’ under topic ‘Monitor’ in the left margin. The monitoring page is shown in Figure 3.



Figure 3: Job monitoring

In this page you will see all your jobs submitted through ECaccess, both those with subscriptions to some events of the operational suite as well as other jobs. You will also see the jobs due for later schedule, as well as those which have already run for the previous notification of some events. Please note that the name of the job is also shown, when available. You can delete jobs from this page. See Changes in job or supression of jobs, below, for more details.

2.2.2. ECaccess Web Toolkit

The same functionality as described above for the web is available through the ECaccess Web Toolkit. These command-line tools are available on the Atos HPC systems at ECMWF. They may also available on your local systems. Please refer to the ECaccess documentation for further details on the ECaccess Web Toolkit - The full featured client.

ecaccess-event-list


The ECaccess Web Toolkit command ecaccess-event-list allows you to list the events available to you.

uid@ac6-200{uid}:1 --> ecaccess-event-list
1247       ai_lwda_00           At this stage, analysis input observations are archived in MARS.
1249       ai_lwda_12           At this stage, analysis input observations are archived in MARS.
1248       ai_oper_00           At this stage, analysis input observations are archived in MARS.
1250       ai_oper_12           At this stage, analysis input observations are archived in MARS.
167        an00h000             At this stage, the analysis at 00UTC is complete.
201        an06h000             At this stage, the deterministic analysis at 06UTC is complete.
168        an12h000             At this stage, the analysis at 12UTC is complete.
202        an18h000             At this stage, the deterministic analysis at 18UTC is complete.
2724       bc_00                 at 00UTC is complete.                  
2725       bc_06                At this stage, the boundary condition forecast at 06UTC is complete.
2726       bc_12                At this stage, the boundary condition forecast at 12UTC is complete.
2727       bc_18                At this stage, the boundary condition forecast at 18UTC is complete.
...

uid@ac6-200{uid}:763 --> ecaccess-event-list an00h000
     Event-id: 167
         Name: an00h000
       Public: yes
        Owner: emos
      Comment: At this stage, the analysis at 00UTC is complete.

Note that either the event number or name can be used with the ECaccess Web Toolkit.

ecaccess-job-submit

When you have found the notification to which you want to attach your job, you can use ecaccess-job-submit to submit your job. This command has been enhanced to allow you to attach your jobs to some event of the ECMWF operational suite. The relevant options for this service are "-eventIds", "-noRenew" and "-manPage".

uid@ac6-200{uid/}:3 --> ecaccess-job-submit -help
Usage:
    ecaccess-job-submit -version|-help|-manual

    ecaccess-job-submit [-debug] [-distant] [-bufsize length]
    [-scheduledDate date] [-noDirectives] [-gateway name] [-remote location]
    [-transferOutput] [-transferError] [-transferInput] [-keep] [-eventIds
    list] [-sterr2Stdout] [-noRenew] [-mailTo email] [-onStart] [-onSuccess]
    [-onFailure] [-onRetry] [-jobName name] [-manPage content] [-lifeTime
    days] [-retryCount number] [-retryFrequency frequency] [-queueName name]
    source

Arguments:
    ...
    -ni, -eventIds list
            Allow giving a list of event-identifiers to subscribe to with
            the Job. The list should be separated by ';' or ','. Only one
            job will be launched when all the events in the list have been
            reached. To submit the same job to multiple events, one will
            need to submit the job to each event separately.
    ...
    -ro, -noRenew
            The job subscriptions to events will not be renewed.
    ...
    -mp, -manPage content
            Allow giving the man page content which will be displayed to the
            ECMWF operators in case of problems with your Job (e.g. what to
            do or who to contact).
    ...

Note that there is no equivalent option with the "ecaccess-job-submit" command to the tick box ”one script to one notification” of the web interface. If you want to submit one job to multiple events, you will have to run multiple ecaccess-job-submit commands.

A sample job submission attached to the event an00h000 could look like follows:

uid@ac6-200{uid/}:4 --> ecaccess-job-submit -queueName ecs -eventIds an00h000 -mp "nothing to be done" -retryCount 1 -retryFrequency 300 job.cmd
6746919

uid@ac6-200{uid/}:5 --> ecaccess-job-list 6746919
Job-Id: 6746919
Job Name: job.cmd
Queue: ecs
Host: ecs-batch.ecmwf.int
Schedule: Feb 04 08:25
Expiration: Feb 11 08:25
Try Count: 0/2
Status: STDBY
Event-Ids: an00h000 (167)

Note the status "STDBY" for the job. This job will remain in standby mode up until the ECMWF operational activity has produced the analysis for the 00Z run of the HRES forecast. Note that the jobs submitted through ECaccess in standby mode will only be visible through ECaccess. They will not be visible using the usual Slurm batch service commands on the Atos HPC.

3. Notes on using the service

3.1. Job status

One advantage of the new service for time-critical jobs submissions under ECaccess is that the ECMWF operators are monitoring your jobs submitted via this system. Also, you can request your jobs to be rerun automatically on failure. However, ECaccess will only be able to show the correct status of your job or possibly rerun your job if it has correctly been notified about the exact status of your job by the Slurm batch service, on the Atos HPC. It is therefore your responsibility to notify correctly the batch service about errors occuring in your job. By default, an error in your job will not be reported to the batch service; the execution of your job will continue and it will finish as if it had completed successfully. One way to stop the execution of your job as soon as there is an error is to use the "set -e" command, in ksh or bash scripts. With this command, your job will stop and exit abnormally as soon as an error occurs. If you want finer control over the errors, you can include some specific tests in your jobs and, for those important tests, exit with a non zero return code, e.g.:

mars request
if [[ $? -ne 0 ]]; then
  echo mars request failed
  exit 1
fi

If you request that your job is restarted after a failure, you will have to make sure that it can be rerun. For example a job doing, e.g.:

set -e
mkdir $SCRATCH/data

cannot be rerun, as the directory $SCRATCH/data will already have been created during the first run. There are different ways to avoid such problems. One way is to switch off "set -e" in some parts of your script, e.g.:

set +e
mkdir $SCRATCH/data
set -e


Another option is to use the conditional execution statement, e.g.:

set -e
mkdir $SCRATCH/data || true


Another way to avoid this particular problem is to work in $SCRATCHDIR as this will be removed when the job ends:

set -e
mkdir $SCRATCHDIR/data

Of course, in this particular example, you could also use the mkdir  " -p " option:

mkdir -p $SCRATCH/data

Other common problems that prevent jobs from being re-runnable are

  • Using ectrans without the "-overwrite" option.  If some files have already been transferred successfully with ectrans then, if the job needs to be re-run, ectrans will fail with the error "eccmd error: Target file already exist (exit=-36)".    To avoid this, use the "-overwrite" option
  • Using bzip2.  By default, bzip2 will not overwrite a compressed file if it already exists.  Use the bzip2 "-f" option to force the file to be overwritten.

Please note that you can submit your job to ECaccess without setting up what is suggested above. Your jobs will run normally but, without this job control, the ECMWF operators will not notice any errors with your jobs and ECaccess will fail to resubmit your jobs, even if you requested some retries.

3.2. Monitoring by the operators

ECMWF operators have a specific interface to monitor user jobs submitted through this system. This allows them to identify various problems, e.g. a general problem with one system at ECMWF or a failure to send a notification to an event, leaving all user jobs waiting to be run. When such problems occur, the operators will try to take corrective action. In case several jobs have failed, apparently linked to a general problem, our operators will be able to restart these jobs, after the problem has been fixed. Our operators will also usually have access to the user's job and job output files, as well as to some instructions you may have provided via the 'man' page, when submitting the job through ECaccess. If no instructions are given, our operators will normally ignore any specific failure of your jobs.

Instructions to the operators should be clear and simple.  Note that our operators will be unable to correct something in your job or under your account; they cannot edit your jobs. Rather then asking the operators to rerun the jobs or to notify someone when a failure occurs, we recommend you to use the automatic job resubmission facility on failure or the email notification option, available with ECaccess or with the batch service.  For example, submitting your job using:

uid@ac6-200{uid/}:5 --> ecaccess-job-submit -queueName ecs -eventIds an00h000 -mp "nothing to be done" -retryCount 3 -retryFrequency 900 job.cmd

will retry your job on failure 3 times with 15 minutes (900 seconds) between retries.  This will sometimes allow the job to complete successfully if the initial failure was caused by a temporary issue.

3.3. Environmental variables

Before submitting the job, the following environment variables are set by ECaccess, are passed to your job and can therefore be used within your job:

  • MSJ_BASETIME  - forecast base time, e.g. "00" or "12"
  • MSJ_STEP            - forecast time step, e.g. "144"
  • MSJ_YEAR           - year of the run, e.g. "2023"
  • MSJ_MONTH      - month of the run, e.g. "02"
  • MSJ_DAY             - day of the run, e.g, "22"
  • MSJ_EXPVER      - version number of data archived in MARS (if relevant), e.g. "0001"
  • MSJ_EVENT        - event name, e.g. "fc00h144"

These environmental variables will help you to access the operational data, e.g. to build the correct date to extract the data from MARS.

3.4. Changes in job or suppression of jobs

If you have to make some changes to any of your ECaccess Time Critical jobs, you will have to cancel the existing job in standby mode and submit the new version of the job. The job name shown with "ecaccess-job- list" or through the web interface should help you to identify the correct job to delete. Similarly, to remove a job from the system, you will have to remove the job in standby (STDBY) mode.

4. Job examples

An example of a Time-critical batch job for submission on the Atos to create various types of ENS meteogram plots is provided by realtime_metgram.sh.

5. Help and Support


For initial help on implementing your jobs in the system or for problems with the "operational" runs of your jobs, please create a Software or Computing ticket in the ECMWF support Portal, providing the Unix user name used and the event your job is submitted to. 

We also recommend that you provide some instructions to our operators in the man page when you submit the job.  But remember that the ECMWF operators will only be able to resubmit your job, not to edit or fix it.