• Created by Unknown User (nagc), last modified on May 25, 2017

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

This tutorial explains how to create a new experiment from an existing one by copying the initial files and setting up the experiment directory.

On this page...

ECMWF HPCF

ECMWF operates a Cray based High Performance Computing Facility (HPCF). It consists of two identical Cray XC40 clusters with their own storage, but with equal access to the high performance working storage of the other cluster. This provides the benefit of having one very large system but the dual clusters add significantly to the resiliency of the system.

Each cluster has 20 cabinets of compute nodes and 13 cabinets of storage. The compute nodes contain two Intel Xeon "Broadwell" processors each with 18 cores. Four compute nodes sit on one blade, sixteen blades sit in a chassis and there are three chassis per frame. This gives a maximum of 192, or 6,912 processor cores per cabinet.

For more details about the ECMWF HPCF, please see: http://www.ecmwf.int/en/computing/our-facilities/supercomputer.

Login to ECMWF Cray High Performance Computing Facility (HPCF)

Each group or participant will have a training user account on the ECMWF system, beginning with 'troifs'. This is different from the user account on the classroom computers.

First login to the ECMWF gateway computer. From here you will login to the HPCF.

ssh troifsXX@ecgb06.ecmwf.int		               # substitute your user id for XX
troifsXX@ecgb06.ecmwf.int's password:              # give your password when prompted using the securID hardware token (one time password)

Next login to the Cray login nodes:

ssh ccb
troifs@ccb-login2:~>

The contents of the account should look like:

troifs1@ccb-login2:~> ls
bin  make  python  t21test  scratch

If any are missing please let us know.

make:  contains the OpenIFS executable for version 40r1. For the purpose of this workshop, the model has already been compiled to avoid delay and adding unnecessary load to the login nodes.

t21test: contains an example low resolution test run for OpenIFS.

Directories bin and python: contains commands and script for use during the workshop.

scratch: is the directory in which we will run OpenIFS and process the output files.

Please do not store large files in your home directory on the gateway or HPCF login.  All OpenIFS experiments and large files should be stored under 'scratch'. The command quota can be used to determine available space.

Creating the experiment initial files

This section will explain how to copy and change the experiment id.

 Initial files for the observed SST experiment, 'ob00', and the climatological experiment 'clim', have been created and are made available in the directory : /perm/rd/openifs/oifs_workshop_2017/expts-inidata/

This directory contains multiple dates for each experiment:

ob00/2015110100 to ob00/2015110100 etc.

Copy previous experiment

Before creating the experiment, we first need to create the initial data. If you plan to rerun the ob00 or clim experiments and use the initial data 'as-is' you can skip this step.

In this example, we will create a new experiment id from the existing ob00 experiment, without changing any of the data itself. You can use your own previous experiments, the only difference is the location of the starting initial data.

In this example, the new experiment is intended for a 50 member ensemble run. We want to use a new experiment id, ob50, to distinguish it from ob00 which only used 10 ensemble members.

First step is to make a copy of the ob00 data to your /scratch directory:

Copy previous initial data from previous experiment...
cd scratch				# starting from your home directory
mkdir -p inidir/ob50 	# our new experiment id
cd inidir/ob50
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110100 .		# those last characters are a 'space' and then a 'fullstop', '.' means "here".

Make sure you use the -L option!  This ensures any 'linked' files are copied as actual files and not symbolic links (true for the fort.4 file). Not doing so will make the fort.4 uneditable.

Ignore any errors about files not being copies because of permission problems.

To copy multiple dates. Example 1: first 5 days
cd inidir/ob50
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110[1-5]* .

Please only copy the initial dates you intend to use. Each date uses 0.5 Gbyte of file storage.

To copy multiple dates. Example 2: copy all dates
cd inidir/ob50
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/201511* .   # this will be slow!

Change the experiment id

OpenIFS forecasts are identified by an 'experiment id', a four letter string. An experiment could consist of a single forecast date or multiple starting dates. It can be of any length and could also include a restarted forecast. However, an experiment only has one horizontal and vertical resolution.

The experiment id is contained the names of the initial files, and it is encoded into the GRIB messages within those files.

The command 'exptid' can be used to check and change the experiment id in the GRIB files.

Confirm experiment id contained in GRIB files..
troifs0@ccb-login3:> exptid ICMSHob00INIT
ICMSHob00INIT, experimentVersionNumber = ob00

The 'experimentVersionNumber' is the GRIB parameter encoded in the GRIB fields.

This can also be seen using the 'grib_ls' command to list the contents of the file:

troifs0@ccb-login3:> grib_ls -p shortName,typeOfLevel,dataDate,experimentVersionNumber ICMSHob00INIT
ICMSHob00INIT
shortName                typeOfLevel              dataDate                 experimentVersionNumber  
t                        hybrid                   20151101                 ob00                    
t                        hybrid                   20151101                 ob00                    
t                        hybrid                   20151101                 ob00  
.......

 

Useful Unix commands

 

Remove a directory and ALL it's contents.. (note this is recursive delete!)
rm -rf 2015110100
rm -rf 2015110[1-5]00
Check size of all files in directory...
du -hs 2015110100
Check filesystem quota (on Cray)..
quota

whicih will produce output similar to the following. The '$SCRATCH' quota is important as this is the total limit for the filesystem /scratch where your experiments are stored.

Quota for $HOME and $PERM:
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa1:/vol/home
                   226M    480M    500M             132   20000   22000        
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa2:/vol/perm
                      0  26624M  27648M               1    200k    210k        

Quota for $SCRATCH ($TEMP) including $SCRATCHDIR ($TMPDIR):
Disk quotas for user troifs0 (uid 16144):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767099824  32212254720 32212254720       -  116570  5000000 5000000       -
Disk quotas for group ectrain (gid 1400):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767105640       0       0       -  118078       0       0       -

 

 

 

  • No labels