The Web Toolkit, previously known as ECaccess shell commands or ECtools, is a set of Perl scripts for the management of files, file transfers, jobs, ectrans associations and events at ECMWF. They can be run by any user and on any Member State host. The Web Toolkit is using SOAP to access the ECaccess web server. Running these commands requires a valid certificate. The command ecaccess-certificate-create will create a certificate in the user's home directory ($HOME/.eccert.crt) from an ECMWF user identifier and a passcode (generated by a security token). You need to ensure the following environment parameters are set with the correct values:
Code Block |
---|
|
http_ecaccess=gateway.meteo.ms:9080 |
...
https_ecaccess=gateway.meteo.ms:9443 |
(e.g. if your local ECaccess Gateway name is "gateway.meteo.ms" and you are using the default ECaccess http/s ports 9080/9443) The default values are pointing to the ecaccessboaccess.ecmwf.int server. Your gateway administrator can provide other default values for these parameters. However, your environment variables take precedence over these default values. If the directory containing the Web Toolkit commands is not in your command path or you do not know the directory in which the commands are installed, try running the "ecaccess-gateway-name" command. If the command is not available, you will need to contact your Computing Representative, your local ECaccess administrator - if known - or User Support at ECMWF. Alternatively, you may wish to install the Web Toolkit yourself (see http://www.ecmwf.int/services/ecaccess/download/see Releases - Web Toolkit). The ECaccess Tools are organized in sets covering access to the whole computing and archiving facilities of ECMWF and are described in the following sections. Each command is documented with its own man page which provide explanation as well as examples on how to use it. The following options are common to all the Web Toolkit commands:
-help | Print a brief help message and exits. |
-manual | Prints the manual page and exits. |
-debug | Display the SOAP messages exchanged. |
-version | Print the ECaccess version number. |
Anchor |
---|
| CertificateManagement |
---|
| CertificateManagement |
---|
|
Certificate management
The ECaccess certificate can be created using the command ecaccess-certificate-create. From an ECMWF user identifier and a PASSCODE (using a security token), it generates a certificate in $HOME/.eccert.crt. Alternatively, a certificate can be created using the ECacess Web interface, see section 7 The Web server. The Web Toolkit is also available at ECMWF. In contrast to using the Web Toolkit installed at your site you will not need a certificate when using it at ECMWF as you have already been validated at login. To display a help screen describing the ecaccess-certificate-create usage:
...
Certificates are PEM/Base64 encoded ASCII files. OpenSSL can be used to decode and display certificate components. The expiry of the various ECaccess services can be displayed with the command ecaccess-certificate-list:
ecgateuid@ad6-101{/home/ectrain/trxuid}:1 1 --> ecaccess> ecaccess-certificate-list
submitJob 168h Jan 18 11:56 job submission
getJobList 168h Jan 18 11:56 job list
deleteJob 168h Jan 18 11:56 delete a job
getJobResult 168h Jan 18 11:56 job result
deleteFile 168h Jan 18 11:56 delete file
getFileList 168h Jan 18 11:56 get file list
mkdir 168h Jan 18 11:56 make directory
getFileSize 168h Jan 18 11:56 get file size
readFile 168h Jan 18 11:56 read file
writeFile 168h Jan 18 11:56 write file
moveFile 168h Jan 18 11:56 move file
rmdir 168h Jan 18 11:56 remove directory
chmod 168h Jan 18 11:56 change file mode
getTempFile 168h Jan 18 11:56 create temporary file
getTransferList 168h Jan 18 11:56 get transfer listsubmitJob 168h Jan 18 11:56 job submission
getJobList 168h Jan 18 11:56 job list
deleteJob 168h Jan 18 11:56 delete a job
getJobResult 168h Jan 18 11:56 job result
deleteFile 168h Jan 18 11:56 delete file
getFileList 168h Jan 18 11:56 get file list
mkdir 168h Jan 18 11:56 make directory
getFileSize 168h Jan 18 11:56 get file size
readFile 168h Jan 18 11:56 read file
writeFile 168h Jan 18 11:56 write file
moveFile 168h Jan 18 11:56 move file
rmdir 168h Jan 18 11:56 remove directory
chmod 168h Jan 18 11:56 change file mode
getTempFile 168h Jan 18 11:56 create temporary file
getTransferList 168h Jan 18 11:56 get transfer list
As can be seen from the output, for a normal user-id the validity is 168 hours (7 days) for all services. The date/time shown refers to the expiration of the certificate.
...
To display the "Message of the day" (/etc/motd on ecgatethe Atos HPC (ecs-login or hpc-login)) containing announcements of upcoming system sessions etc., use the command ecaccess-cosinfo:
-> ecaccess-cosinfo
*******************************************************************************
* *
* *
* Welcome to AIX Version 5.3! *
* *
* *
* Please see the README file in /usr/lpp/bos for information pertinent to *
* this release of the AIX Operating System. *
* *
* *
*******************************************************************************
ECMWF SYSTEM SESSIONS> ecaccess-cosinfo
ECMWF SYSTEM SESSIONS
--------------------
WEDNESDAY the 12th of JANUARY 2011WEDNESDAY the 25th of JANUARY 2023:
----------------------------------
08:30-10:30 UTC Mars and ECFS System Session30 UTC Mars and ECFS System Session:
impact: Mars and ECFS will be unavailable
To reread the message please use: more /etc/motd or cat Mars and ECFS will be unavailable
To reread the message please use: more /etc/motd or cat /etc/motd
==============================================================================
...
Commands named ecaccess-gateway-* (table 2) provide information about ECaccess gateways (see section 2.1). Table 2: Web Toolkit commands providing information on ECaccess gateways (ecaccess-gateway-*). ECaccess Gateway), see following table:
The command ecaccess-gateway-name is the only one which can be used without authentication/certificate.
...
Anchor |
---|
| FileManagement |
---|
| FileManagement |
---|
|
File managementFiles at ECMWF can be managed through the Web Toolkit commands named ecaccess-file-*. The file location is specified with the following syntax [domain:][/user-id/]path where domain: can take the values as listed in table 3. The user-id refers to a an ECMWF computer user-id. Only if the domain is specified as an ECFS domain, then the user-id could also be a common pool, e.g. demeter. Table 3: Domain values.
Domain value | Purpose |
home: | the |
ecgate Atos HPC $HOME file system |
scratch: | the |
ecgate Atos HPC $SCRATCH file system |
ec: | the ECFS domain ec: |
ectmp: | the ECFS domain ectmp: |
host-name: | any server at ECMWF, e.g. |
c2aIf no domain is specified then an absolute path will translate to an absolute path on the ecgate server Atos HPC and a relative path will translate to a path in the HOME directory of the current user. If no user-id is specified then the current user-id is selected by default. When you specify a host-name you are by default under the root directory; the user-id parameter can not be used with a host-name. File location examples:
bin/a.out | file a.out in directory $HOME/bin of the current user |
home:bin/a.out | file a.out in directory $HOME/bin of the current user |
/tmp/a.out | file a.out in directory /tmp on |
ecgateecs-login |
home:/xzy/bin/a.out | file a.out in directory $HOME/bin of user xzy |
ec:bin/a.out | file a.out in directory bin in the current user's ECFS domain ec: |
ec:/xzy/bin/a.out | file a.out in directory bin in user xzy's ECFS domain ec: |
c2ac2atmp/groupvol/xzy/a.out | file a.out in directory / |
c2atmp/systems c2aTable 4: Web Toolkit commands for file management (ecaccess-file-*).
...
Batch job management
Batch jobs at ECMWF can be managed through the Web Toolkit commands named ecaccess-job/queue-* , see table 5. Possible values of the job status, which can be checked with the command ecaccess-job-list or via the Web interface, are listed in table 6. A special service (see option -eventIds) allows to automatically submit jobs to be run when certain points in the daily ECMWF operational forecast suite have been reached. The main purpose is to ensure that certain data are available before e.g. submitting a MARS request. These events correspond to the different stages when the ECMWF operational activity has produced certain data or products. The list of events can be retrieved with the "ecaccess-event-list" command. Table 5: Web Toolkit commands for batch job management (ecaccess-job/queue-*).in the table below.
Table 6: Job statusPossible values of the job status, which can be checked with the command ecaccess-job-list or via the Web interface, are listed in table below.
Status | Meaning |
DONE | Jobs have successfully completed |
EXEC | Jobs are running |
INIT | Jobs are being initialised |
RETR | Jobs will be resubmitted |
STDBY | Jobs are waiting for an event |
STOP | Jobs have NOT completed (error) |
WAIT | Jobs have been queued to the scheduler (e.g. LoadLeveler) |
...
A special service (see option -eventIds) allows to automatically submit jobs to be run when certain points in the daily ECMWF operational forecast suite have been reached. The main purpose is to ensure that certain data are available before e.g. submitting a MARS request. These events correspond to the different stages when the ECMWF operational activity has produced certain data or products. The list of events can be retrieved with the "ecaccess-event-list" command.
Management of events
ECMWF maintains some notifications (events) which are linked to ECMWF's operational activity and offers the service for time-critical jobs (see also separate documentation). This service is also available to MS users who maintain their own notifications and can therefore create simple dependencies between different activities, at ECMWF and remote sites. The shell commands to managed events are listed in table 7. Table 7: Web Toolkit commands for management of events at ECMWF (ecaccess-event-*).
...
Management of ECtrans transfers
The commands for the management of ECMWF-initiated transfers (ectrans, see section 4.2) are listed in table 8the table below. They can only be used for the management of transfers, which have used the ECaccess gateway as shown with the ecaccess-gateway-name command. Table 8: Web Toolkit commands for management of ECMWF-initiated transfers (ecaccess-ectrans-*).
...
Anchor |
---|
| ManagementOfEctransAssociations |
---|
| ManagementOfEctransAssociations |
---|
|
Management of ECtrans associationsBefore making use of ectrans, users will need to declare an ectrans association (previously referred to as 'remote Member State user (msuser)') for the storage/retrieval of the remote file. The management of these associations can be done through the Web Toolkit commands named ecaccess-association-* listed in table 9. Table 9: Web Toolkit commands for management of ECtrans associations (ecaccess-association-*).the table below.
...
Execution return codes
The option '-debug' can be used with any Web Toolkit command to display information concerning the SOAP messages exchanged. If a Web Toolkit command doesn't work correctly, please run it with the -debug option and send the output to ECMWF for further investigation. Web Toolkit commands return 0 if successful, and a non-zero code otherwise. Each time an error occurs, a message indicating the error is displayed to the user.