• Created by Unknown User (nagc), last modified on May 25, 2017

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

This tutorial explains how to create a new experiment from an existing one by copying the initial files and setting up the experiment directory.

On this page...

ECMWF HPCF

ECMWF operates a Cray based High Performance Computing Facility (HPCF). It consists of two identical Cray XC40 clusters with their own storage, but with equal access to the high performance working storage of the other cluster. This provides the benefit of having one very large system but the dual clusters add significantly to the resiliency of the system.

Each cluster has 20 cabinets of compute nodes and 13 cabinets of storage. The compute nodes contain two Intel Xeon "Broadwell" processors each with 18 cores. Four compute nodes sit on one blade, sixteen blades sit in a chassis and there are three chassis per frame. This gives a maximum of 192, or 6,912 processor cores per cabinet.

For more details about the ECMWF HPCF, please see: http://www.ecmwf.int/en/computing/our-facilities/supercomputer.

Login to ECMWF Cray High Performance Computing Facility (HPCF)

Each participant will have a training user account on the ECMWF system. This is different from the user account on the classroom computers.

First login to the ECMWF gateway computer. From here you will login to the HPCF.

ssh troifsXX@ecgb06.ecmwf.int		               # substitute your user id for XX
troifsXX@ecgb06.ecmwf.int's password:              # give your password when prompted using the securID hardware token (one time password)

Next login to the Cray login nodes:

ssh ccb
troifs@ccb-login2:~>

The contents of the account should look like:

troifs1@ccb-login2:~> ls
bin  make  python  t21test  scratch

If any are missing please let us know.

Directory make:  contains the OpenIFS executable for version 40r1. For the purpose of this workshop, the model has already been compiled to avoid delay and adding unnecessary load to the login nodes.

Directory t21test: contains an example low resolution test run for OpenIFS.

Direcories bin and python: contains commands and script for use during the workshop.

Directory scratch: is the directory in which we will run OpenIFS and process the output files.

Please do not store large files in your home directory on the gateway or HPCF login.  All OpenIFS experiments and large files should be stored under 'scratch'. The command quota can be used to determine available space.

Creating the experiment initial files

This section will explain how to copy and change the experiment id.

 Initial files for the observed SST experiment, 'ob00', and the climatological experiment 'clim', have been created and are made available in the directory : /perm/rd/openifs/oifs_workshop_2017/expts-inidata/

This directory contains multiple dates for each experiment:

ob00/2015110100 to ob00/2015110100 etc.

Copy previous experiment

Before creating the experiment, we first need to create the initial data. If you plan to rerun the ob00 or clim experiments and use the initial data 'as-is' you can skip this step.

In this example, we will create a new experiment id from the existing ob00 experiment, without changing any of the data itself. You can of course, use your own previous experiments, the only difference is the location of the starting initial data.

In this example, the new experiment is intended for a 50 member ensemble run. We want to use a new experiment id, ob50, to distinguish it from ob00 which only used 10 ensemble members.

First step is to make a copy of the ob00 data to your /scratch directory:

Copy previous initial data from previous experiment...
cd scratch				# starting from your home directory
mkdir -p inidir/ob50 	# our new experiment id
cd inidir/ob50
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110100 .

Make sure you use the -L option!  This ensures any 'linked' files are copied as actual files and not symbolic links (true for the fort.4 file). Not doing so will make the fort.4 uneditable.

Ignore any errors about files not being copies because of permission problems.

To copy multiple dates. Example 1: first 5 days
cd inidir/ob50
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110[1-5]* .

Please only copy the initial dates you intend to use. Each date uses 0.5 Gbyte of file storage.

To copy multiple dates. Example 2: copy all dates
cd inidir/ob50
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/201511* .   # this will be slow!

Useful Unix commands

 

Remove a directory and ALL it's contents.. (note this is recursive delete!)
rm -rf 2015110100
rm -rf 2015110[1-5]00
Check size of all files in directory...
du -hs 2015110100
Check filesystem quota (on Cray)..
quota

whicih will produce output similar to the following. The '$SCRATCH' quota is important as this is the total limit for the filesystem /scratch where your experiments are stored.

Quota for $HOME and $PERM:
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa1:/vol/home
                   226M    480M    500M             132   20000   22000        
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa2:/vol/perm
                      0  26624M  27648M               1    200k    210k        

Quota for $SCRATCH ($TEMP) including $SCRATCHDIR ($TMPDIR):
Disk quotas for user troifs0 (uid 16144):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767099824  32212254720 32212254720       -  116570  5000000 5000000       -
Disk quotas for group ectrain (gid 1400):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767105640       0       0       -  118078       0       0       -

 

 

 

  • No labels