Cray's Performance Analysis Tool (CrayPAT) can be used to investigate the performance of OpenIFS on Cray supercomputers. It can provide very detailed information on the timing and performance of the code.

Using it with OpenIFS is straightforward but may require a small change to the fcm configuration file.

fcm is the name of the command and software used to compile OpenIFS.

Step-by-step guide

On a Cray system, to use CrayPAT with OpenIFS:

  1. Ensure that the perftools module is loaded before compiling any OpenIFS code. This is important because perftools modifies each object (.o) file for the subsequent analysis by CrayPAT.

    module load perftools

  2. The normal behaviour of the OpenIFS compilation using the fcm command is to perform the final link step by creating and linking against a temporary archive library (libmaster.a) of all the object (.o) files. This temporary library by default will go into a directory specified by the shell environment variable TMPDIR. The location of this temporary library can be altered by changing TMPDIR.
    However, this behaviour is not suited to CrayPAT which requires all object files to still be available after compilation and errors similar to the following will occur:

    Possible errors from compilation
    [FAIL] WARNING: CrayPat is saving object files from a temporary directory into directory '/home/user/.craypat/master.exe/3350'
[FAIL] cp: invalid option – '/'
[FAIL] Try `cp --help' for more information.
[FAIL] basename: invalid option – 'L'
[FAIL] Try `basename --help' for more information.
[FAIL] WARNING: the link is executed in a temporary directory which may not be accessible later for pat_build

  3. To overcome this, add a new configuration option to the OpenIFS fcm configuration file: make/oifs.cfg

    oifs.prop{keep-lib-o} = true


    This line can be added anywhere in the file but perhaps is best placed after the line oifs.target = .
    This option changes the behaviour of fcm so that a new directory 'lib' will be created in the build directory (typically cce-opt/oifs/lib) and the archive library for the compiled programs will be stored in there suitable for later use by CrayPAT.

  4. Make a clean compilation of OpenIFS, ensuring all code is recompiled. Either use the --new option to the fcm-make command: 

    fcm make --new ...

    or delete the entire build directory (e.g. cce-opt).

    The resulting executable binary should then be suitable for use with pat_build e.g.  
       

    pat_build -w make/cce-opt/oifs/bin/master.exe

  5. Using the CrayPATH API  (pat_record) is a good way to avoid the large startup cost of the model on the first timesteps.  A good way to start is adding calls to cnt4.F90 around the main time-stepping loop. For the CY38R1 version of the model, this starts at line 483 in cnt4.F90 and ends at line 1360.  It's recommended that code is added to avoiding recording until after a few timesteps have completed to ensure everything in the model has been invoked at least once (e.g. radiation).

     

For more information on CrayPAT, refer to Cray's Perftool documentation.

For more information about FCM and the 'fcm make' options, see the FCM User Guide.

For further assistance with optimizing OpenIFS, please contact OpenIFS support: oifs-support@ecmwf.int. ECMWF have extensive experience of optimizing the IFS model and can provide advice and assistance.