If you wish to use ecFlow to run your workloads, ECMWF will provide you with ready-to-go ecFlow server running on an independent Virtual Machine outside the HPCF. Those servers would take care of the orchestration of your workflow, while all tasks in your suites would actually be submitted and run on HPCF. With each machine being dedicated to one ecFlow server, there are no restrictions of cpu time and no possibility of interference with other users.

Getting started

If you don't have a server yet, please raise an issue through the ECMWF support portal requesting one.

We will create one for you and give you the server details, so you can connect straightaway from both your ecFlow client on the command line as well as the ecFlow UI.

For example, if you are given the server ecflow-gen-$USER-001, you can check it's up and running and see some stats from the HPCF with:

$ module load ecflow
$ export ECF_HOST=ecflow-gen-$USER-001                                                                                                                                
$ ecflow_client --ping
$ ecflow_client --stats

To configure it on your ecFlow UI, just go into Servers-Manage Servers and click on Add Server.

You can then fill in the form with the Host given and Port 3141.

ssh $ECF_HOST sudo systemctl restart ecflow-server

4.4.14
user
-*

ecflow_client --reloadwsfile

Preparing your suites and tasks

Both HPCF and ecFlow Server will recognise you as the same user and use the same HOME and PERM filesystems. For simplicity, in most cases the easiest solution is to keep your suite's ECF_HOME, ECF_FILES, ECF_INCLUDE on HOME or PERM, instead SCRATCH or HPCERM.

While the availability of virtual infrastructure to run ecFlow servers remains limited, you may start your ecFlow servers in the interim HPCF dedicated node to be able to run your suites. 

At a later stage, those ecFlow servers will need to be moved to dedicated Virtual Machines outside the HPCF, where practically no local tasks will be able to run. All ecFlow tasks will need to be submitted to one of the HPCF complexes through the corresponding Batch system.

Please do keep that in mind when migrating or designing your solution.

Starting the ecFlow server

The server needs to be started using the usual procedure on one of the AA login nodes, not through an interactive job.

module load ecflow troika
ecflow_start.sh <options>

You may wish to pass extra options to configure the port or your ecflow home. 

...

Remember that all tasks will need to be submitted as jobs through the batch system, so you should avoid running tasks locally on the node where the server runs. Make sure that your task header contains the necessary SBATCH directives to run the job. As a minimum:

...

Example of task include files enabling communication between a batch job and ecFlow servers are available from ECMWF git repository.

Logservers

If you decide to store the jobs standard output and error on a filesystem only mounted on the HPCF (such as SCRATCH or HPCPERM), your ecFlow UI running outside the HPCF - such as your VDI, will not be able to access the output of those jobs out of the box. In that case you would need to start a log server on the Atos HPCF so your client can access those outputs. The logserver must run on the hpc-log node, and if you need a crontab to make sure it is running you should place it on hpc-cron.

Trapping of errors

It is crucial that ecFlow knows when a task has failed so it can report accurately what is the state of all your tasks in your suites. This is why you need to make sure error trapping is done properly. This is typically done in one of your ecFlow headers, for which you have an example in the  ECMWF git repository.

Job management

ecFlow delegates the job management tasks such as submission, kill or monitor the status to external applications. For your convenience, you may use troika, a tool that will take care of those tasks. To use it, just make sure you have the following variables defined at the suite level:

edit QUEUE nf
edit SCHOST aahpc
edit ECF_JOB_CMD troika submit -o %ECF_JOBOUT% %SCHOST% %ECF_JOB%
edit ECF_KILL_CMD troika kill %SCHOST% %ECF_JOB%
edit ECF_STATUS_CMD troika monitor %SCHOST% %ECF_JOB%

Of course, you may change queue to np if you are running bigger parallel jobs, or SCHOST to eventually run on other complexes other than aa. 

By default scancel doesn't send signals other than SIGKILL to the batch step. Consequently, one should use "-b" or "-f" option to send a signal ecFlow job is designed to trap before notifying ecFlow server the job was killed:

scancel --signal=TERM -b ${SLURM_JOB_ID}

In this example, SIGTERM (15) was sent but one can use other signals as well. This option can also be used with Troika to kill jobs from ecFlow_ui. This can be specified in the configuration file such as this one:

To use a custom troika executable or personal configuration file with Troika, ecFlow variables should be defined like this:

edit QUEUE nf
edit SCHOST aahpc
edit TROIKA /path/to/bin/troika
edit TROIKA_CONFIG /path/to/troika.yml
edit ECF_JOB_CMD troika%TROIKA% -c {PATH}/troika.yml%TROIKA_CONFIG% submit -o %ECF_JOBOUT% %SCHOST% %ECF_JOB%
edit ECF_KILL_CMD troika%TROIKA% -c {PATH}/troika.yml%TROIKA_CONFIG% kill %SCHOST% %ECF_JOB%
edit ECF_STATUS_CMD troika%TROIKA% -c {PATH}/troika.yml%TROIKA_CONFIG% monitor %SCHOST% %ECF_JOB%

Connecting to the ecFlow server 

Due to the current limitation in network connectivity to arbitrary ports between our Reading and Bologna Data Centres, it is not possible to connect to that ecflow server in AA from your usual ecflow_ui in Reading.

...

You are of course free to use any other solution for the job submission, kill and monitor of your jobs. If you write your own, please note that:

• You will need to use ssh "<complex>-batch" to run the relevant slurm commands on the appropriate complex.

• If the job has not started when the kill is issued, your ecFlow server would not be notified that the job has been aborted. You would need to manually set it to aborted, or alternatively use -b or -f options so that scancel sends the signal once the job has started.
  

  No Format
  scancel --signal=TERM -b ${SLURM_JOB_ID}

  
  

Connecting to the ecFlow server 

Through VDI

ecflow UI is installed in your VDI, and you can use it to connect to your ecFlow server straight away. 

Through a graphical VNC session

You may spin up a graphical VNC session on the HPCF with ecinteractive on your VDI. Once in the VNC session, you can then do the following from a terminal within that VNC session:

...

You may alternatively use the native ecflow_ui client in your End User Device or VDI, but an additional step is required to ensure connectivity between both ends. You will need to create an SSH tunnel, forwarding the port where the ecflow server is running. 

1. Authenticate via Teleport on your End User device

2. Create the SSH tunnel with:

3. Start your ecflow server with your preferred settings on one of the login nodes of AA with ecflow_start.sh

4. Once you know the hostname and port of the server, from your Linux Desktop or VDI create the SSH tunnel

   No Format
   ssh -N -L<ecflow_port>L3141:localhost:<ecflow_port>3141 -J jump.ecmwf.int,hpc-login <ecflow_host>

   
   

   where the first '3141' is the local port. For example, if the server is started on the host aa6-100, port 34567 ecflow-gen-user-001:

   No Format
   ssh -N -L34567L3141:localhost:34567 aa6-1003141 -J jump.ecmwf.int,hpc-login ecflow-gen-user-001

   
   

5. Open ecflow_ui on your End User Device or VDI and configure the new server, using "localhost" as the host and the ecflow ecFlow (local) port used above.

X11 forwarding

You may also run ecflow_ui remotely on the Atos HPCF, and use X11 forwarding to display on your screen:

ssh -X aahpc-login
module load ecflow
ecflow_ui

In this case, when adding the server remember it needs to be configured with the real name of the host running the ecflow server, e.g. ecflow-gen-$USER-001.