HPC2020: How to connect

From outside ECMWF, you may use Teleport through our gateway in Bologna, jump.ecmwf.int. Direct access through ECACCESS service is not available.

$> tsh login --proxy=jump.ecmwf.int
$> ssh -J user@jump.ecmwf.int user@hpc-login
# or for users with no formal access to HPC service:
$> ssh -J user@jump.ecmwf.int user@ecs-login

Atos HPCF: System overview

The Atos HPCF consists of four virtually identical complexes: AA, AB, AC and AD. In total, this HPCF features 8128 nodes:

• 7680 compute nodes, for parallel jobs
• 448 GPIL (General Purpose and Interactive Login) nodes, which are devised to integrate the interactive and post-processing work from older platforms such as the Cray HPCF, ECGATE and Linux Clusters.

HPC2020: Shells

You will find a familiar environment, similar to other ECMWF platforms. Bash and Ksh are available as login shells, with Bash being the recommended option.

Note that CSH is not available. If you are still using it, please move to a supported shell.

HPC2020: Filesystems

The filesystems available are HOME, PERM, HPCPERM and SCRATCH, and are completely isolated from those in other ECMWF platforms in Reading such as ECGATE or the Cray HPCF.

Filesystems from those platforms are not cross-mounted either. This means that if you need to use data from another ECMWF platform such as ECGATE or the Cray HPCF, you will need to so transfer it first using scp or rsync. See HPC2020: File transfers for more information.

HPC2020: File transfers

For transfers to ECMWF, we recommend using rsync which will transfer the files over an ssh connection. For that, you will need to have Teleport configured with the appropriate settings in your ssh config file.

Any file transfer tool that supports SSH and the ProxyJump feature should work, such as the command line tools sftp or scp. Alternatively, you may also use the Linux Virtual Desktop and its folder sharing capabilities to copy local files to your ECMWF's HOME or PERM.

HPC2020: Software stack

A number of software, libraries, compilers and utilities are made available through the HPC2020: The Lmod Module system.

If you want to use a specific software package, please check if it is already provided in modules. If a package or utility is not provided, you may install it yourself in your account or alternatively report as a "Problem on computing" through the ECMWF Support Portal.

HPC2020: Batch system

HPC2020: Cron service

If you need to run a certain task at given regular intervals automatically, you may use our cron service available on the hosts hpc-cron for HPC users and ecs-cron for those with no access to the HPCF.

HPC2020: Using ecFlow

If you wish to use ecFlow to run your workloads, ECMWF will provide you with ready-to-go ecFlow server running on an independent Virtual Machine outside the HPCF. Those servers would take care of the orchestration of your workflow, while all tasks in your suites would actually be submitted and run on HPCF. With each machine being dedicated to one ecFlow server, there are no restrictions of cpu time and no possibility of interference with other users.

HPC2020: Accounting

To ensure that computing resources are distributed equitably and to discourage irresponsible use users' jobs on the Atos HPCF are charged for the resources they have used against a project account. Each project account is allocated a number of System Billing Units(SBU) at the beginning of each accounting year (1 January to 31 December). You can monitor your usage in the HPC SBU accounting portal.

HPC2020: ECacccess

ECaccess in Bologna does not offer interactive login access any longer. Users will either use teleport (Teleport SSH Access) or VDI (How to connect - Linux Virtual Desktop VDI) or access the Atos systems in Bologna.

The ECACCESS web toolkit services, such as the job submission, including Time-Critical Option 1 jobs (see below), file transfers and ectrans have been set up on Atos HPCF with the ECACCESS gateway boaccess.ecmwf.int. Previously installed remote (at your site) ECaccess Toolkits should be able to interact with this new gateway. However, we would recommend you to install the latest version, available from Releases - Web Toolkit. To make the remote existing ECaccess Toolkits working with the ECaccess gateways in Bologna, users will need to define the following 2 environmental variable, e.g. to talk to the ECMWF ECaccess gateway in Bologna:

HPC2020: Time Critical option 1 activities

The ECaccess software includes the service of launching user jobs according to the dissemination schedule (Dissemination schedule) of ECMWF's real-time data and products. This service is also known as TC-1 service, or TC-1 jobs. For more information on TC-1, see Simple time-critical jobs.

HPC2020: Time Critical Option 2 setup

Under the Framework for time-critical applications Member States can run ecFlow suites monitored by ECMWF. Known as the option 2 within that framework, they enjoy a special technical setup to maximise the robustness and high availability similar to ECMWF's own operational production. When moving from a standard user account to a time-critical one (typically starting with a "z" followed by two or three characters) there are a number of things you must be aware of.

HPC2020: GPU usage for AI and Machine Learning

The Atos HPCF features 26 special GPIL nodes with GPUs for experimentation and testing for GPU-enabled applications and modules, as well as Machine Learning and AI workloads. Present in only one of the complexes (AC), each node is equipped with 4 NVIDIA A100 40GB cards. They can be used in batch through the special "ng" QoS in the SLURM Batch System. Interactive jobs are also possible with ecinteractive -g.

HPC2020: Missing features and known issues

Atos HPCF is not operational platform yet, and many features or elements may be gradually added as complete setup is finalised. Here is a list of the known limitations, missing features and issues.

HPC2020: FAQs

Here are the most common pitfalls users face when working on our Atos HPCF.

• HPC2020: Reading a NetCDF or HDF5 file gets stuck
• HPC2020: EcFlow fails to submit any tasks