ECaccess and ECaccess Web Toolkit tutorial

Let us explore the ECaccess Web Toolkit and ECaccess services. The web Toolkit can be installed on your local system; it is also available at ECMWF on ECS or HPC.

Availability of Web Toolkit

1. Is the Toolkit available to you?

   Solution - At ECMWF

   In a terminal on ECS or HPC, try the following commands:

   ecaccess
   # you may need to load the ecaccess module:
   module load ecaccess
   # then try again:
   ecaccess

   Solution - At your local site

   In a local terminal or on some of your servers, try the following command:

   ecaccess

   If the command is not found, you may need to update the PATH to point to the Web Toolkit, or may have to install the Toolkit locally.

2. Optional: if you use the Web Toolkit at your local site, you will first need to create a valid certificate:

   Answer - At your local site only

   In a local terminal at your local site, try to run the command:

   ecaccess-certificate-list

   Either you will see the duration of validity of the different components of the Toolkit or you will be informed that there is no valid certificate found and you will be asked to create one:

   ecaccess-certificate-create

   Note that you will need to use the TOTP service to authenticate yourself to generate the Toolkit certificate.

ECaccess interactive sessions

1. Where do you access the ECaccess web interface?

   Solution - If you have a local ECaccess gateway

   In a terminal session on ECS or HPC at ECMWF, you can run the command:

   ecaccess-gateway-list

   This will show you all the ECaccess gateways installed. If you can see one ECaccess gateway installed at your local site, you can point your local browser to its URL.

   Solution - If no local gateway available

   You can access the ECaccess web interface on http://boaccess.ecmwf.int/

2. How to access ECMWF via ECaccess through sftp

   Solution - If you have a local ECaccess gateway

   In a local terminal session at your site you can run the command:

   sftp <your_ecmwf_user>@<gateway_name>

   You will see the different ECMWF 'file systems', a.k.a. domains, available to you.

   Solution - If no local gateway available

   In a local terminal session at your site you can run the command:

   sftp <your_ecmwf_user>@boaccess.ecmwf.int

   You will see the different ECMWF 'file systems', a.k.a. domains, available to you.

ECaccess file management

1. How to list, upload, remove a file in $SCRATCH

   Solution - you will more likely use this option on a local system at your site.

   In a terminal session, you can run the commands:

   cd
   ecaccess-file-dir scratch:
   ecaccess-file-put <local_file> scratch:
   ecaccess-file-dir scratch:<local_file>
   ecaccess-file-delete scratch:<local_file>
   ecaccess-file-dir scratch:<local_file>

   where <local_file> is any file available in your current working directory.

ECaccess job management

1. How to submit, monitor, cancel a job through ECaccess

   Solution - again, you are more likely to use this option on a local system at your site.

   In a terminal session, you can run the commands:

   ecaccess-queue-list
   ecaccess-queue-list ecs
   ecaccess-job-submit -help
   ecaccess-job-submit -queue ecs my_slurm_job.cmd
   ecaccess-job-list [ECaccess_Job_Id]
   ecaccess-job-delete <ECaccess_Job_Id>

   Where 'my_slurm_job.cmd' is a SLURM job in your current working directory and ECaccess_Job_Id is the ECaccess job ID returned to you when you submitted the job.

   Note that ECaccess has its own batch job scheduling system. It can submit a job to different computing systems, possibly running different batch workload managers. The command 'ecaccess-queue-list' will show you which computing systems and which batch workload managers are available.

Ectrans - unattended file transfers

1. How to setup an ectrans association?

   Solution - Setting un the ectrans association

   In a terminal session, either at your local site, if the web Toolkit is installed, or at ECMWF on ECS or HPC, you can run the commands:

   ecaccess-association-list
   ecaccess-association-get -template ectrans_ecmwf_$USER ectrans_ecmwf_$USER.txt
   vi ectrans_ecmwf_$USER.txt

   You will have to update the main parameters given below, to point to the destination system where you wnat to transfer files to, e.g.:

   ##############################################################
   # Main Parameters
   ##############################################################
   $name='ectrans_ecmwf_<user>';
   $active='yes';
   $comment='transfers from ecmwf';
   $grantedUserList='';
   $directory='./files_from_ecmwf';
   $hostName='my_local_hostname.meteo.ms';
   $login='my_local_user';
   $protocol='genericFtp';

   Note that the login requested is your login on the system (hostName) where you will transfer files to.

   The Protocol will usually be genericFtp or genericSftp. Once the association is defined, you should push it to the ECaccess gateway with, e.g.:

   ecaccess-association-put -password ectrans_ecmwf_$USER.txt
   ecaccess-association-list

2. How to use ectrans?

   Solution - Using ectrans

   In a terminal session, on ECS or HPC, you can run the commands:

   ectrans -help
   ectrans -remote ectrans_ecmwf_$USER -source <local_file> -verbose
   ecaccess-ectrans-list [Ectrans_Id]

   where Ectrans_Id is the ectrans Identifier returned to you, when running ectrans (in asynchronous mode). If the transfer fails, see for what reason it failed. You may need to update the association used and push it back to ECaccess, before restarting the ectrans transfer:

   ecaccess-ectrans-restart <Ectrans_Id>

ECaccess time Critical option 1 jobs

1. How to submit a TC-1 job?

   Solution - Submitting a TC-1 job

   In a terminal session, either at your local site, if the web Toolkit is installed or at ECMWF, on ECS or HPC, you can run the commands:

   ecaccess-event-list
   ecaccess-event-list ef12h240metgram

   At ECMWF, you can take a copy of all the sample SLURM jobs, including the realtime meteogram job, with:

   git clone ssh://git@git.ecmwf.int/uss/example_jobs.git
   vi example_jobs/atos_examples/realtime_metgram.cmd

   You should adapt the sample job to your requirements. Note that you will have to update the SLURM '#SCRATCH -chdir=' directive. You can now submit the job, e.g. with:

   ecaccess-job-submit -queueName ecs -eventIds ef12h240metgram example_jobs/atos_examples/realtime_metgram.cmd
   ecaccess-job-list

   The TC-1 job just submitted should be in 'Standby' mode. It will run when the 10-days realtime ENS meteograms for the 12Z run are available. 

   If you no longer want one of your TC-1 jobs to run or if you need to make a correction to the job, you will have to remove the relevant ECaccess job in Standby mode, with 'ecaccess-job-delete'. If wanted, you can then adapt the job and resubmit it.