In this article we explain how to use OpenIFS in a container environment. At present this has been tested in two different ways using Docker on a Linux workstation:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
Motivation
Setting up the computing environment (the libraries, directory structure, etc) required by OpenIFS can present a challenge when it is necessary to run the model on a different hardware infrastructure, for instance during workshops and training events. It is time consuming to install and compile the model and all of its required software packages. Also, the libraries that are available on the local system may not be compatible with the model requirements.
Many of these issues can be avoided by running a containerised version of the model which is a self-sufficient code package that can be used in a consistent way on different hardware platforms. The computational overhead (the "costs") of the container environment itself is often outweighed by performance increases due to the local availability and the instant access to all required libraries and data within the container.
We have used the Docker platform to produce a container image for OpenIFS. This requires the design of a "Dockerfile" which describes the build process for the model code and all its dependent libraries, and which results in a binary Docker image. This image can be uploaded onto other computers that make use of the Docker platform or which use other compatible software. A “container” is the running instance of the Docker image.
Pre-compiled OpenIFS Docker images can be made available for download from container image repositories (e.g. Docker Hub or Harbor) and should be able to run on any computer that uses the Docker platform without the need to install and compile the model or any additional software. The OpenIFS Docker images should also work with other container software compatible with the Open Container Initiative (OCI) standards such as Singularity or Sarus.
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
|
Dockerfiles
The Dockerfile describes the build process of the container image. Examples for several of these files are provided in the OpenIFS git repository. The naming convention for Dockerfiles is as such:
| Panel | ||||
|---|---|---|---|---|
| ||||
Dockerfile.oifs<MODELRELEASE>.<GITHASH>.<NOTE>.<ARCH>.<TYPE> MODELRELEASE: A string generated from the IFS cycle, release number and OpenIFS version , e.g. 40r1v2 for CY40R1 OpenIFS release v2. GITHASH: The first five characters of the OpenIFS repository git commit from which this image is built. NOTE: An optional comment string that describes features of this build. ARCH: The architecture for which this image is built, e.g. x86_64, amd64, i386 etc. TYPE: dev, test or bld. bld (build) to be used only for production images, such as from existing OpenIFS releases. dev should be used for images created from development branch commits. |
Example: Dockerfile.oifs40r1v2.41537.user.x86_64.bld
Example Build Process
This section describes the generation process of a container image from the Dockerfile. You need the following files in your build directory:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
The Dockerfile can be obtained from the git repository. As described through the naming convention it will generate an image of OpenIFS 40r1 version 2. The partial git hash relates the Dockerfile (and image) to a specific git commit in the OpenIFS repository (in this case it relates to 415374d which is tagged as model release v2 ). The note 'root' indicates that when the image is loaded as a container the user will have root privileges. This will allow us to explore the directory structure of the container image. For convenience it is recommended to create a symbolic link to the generic name Dockerfile.
The tar archive oifs40r1v2.41537.tar.gz is created from the model sources after they have been checked out from the git repository (again the partial commit is specified). The Dockerfile will expect the tar archive in the same directory and the file name is specified.
The following command builds the image oifs40r1v2.415374.root. The generic command is docker build -t <image_name> however at ECMWF four variables need to be set for network proxies in order to access the internet from within the container.
| Panel | ||||
|---|---|---|---|---|
| ||||
|
This starts the generation process of the image which contains the minimum of software that is required to run OpenIFS. The image is based on a Ubuntu Linux LTS version and in a first step the necessary developer tools are installed (e.g. GNU compiler, MPI and maths libraries). Afterwards the ecCodes library is obtained via download from the web and compiled with minimum settings. Then the OpenIFS sources are added from the tar archive, required environment variables are set and the model binaries are compiled. During a last step various file permissions are set and the model executable is moved to a globally accessible location.
At the end of the build process the successful image creation is shown as: Successfully tagged oifs40r1v2.415374.root:latest
| Info |
|---|
Dockerfiles with the note ' |
Now we can verify the that the image is available and we will load it into a container using the the docker run command:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
Our command line prompt has changed as we are now 'root' inside the container. A file listing shows the directory structure inside the container.
| Panel | ||||
|---|---|---|---|---|
| ||||
root@38b1649e05b9:/# ls -F |
The OpenIFS model is installed in /oifs. The ecCodes library is found in its default destination under /usr/local/lib.
In order to run the acceptance test the file /oifs/t21test/job needs editing:
EXPID=epc8MASTER=/usr/local/bin/master.exe
When setting GRIB_SAMPLES_PATH replace grib_api with eccodes.
In order to run the executable with mpirun as root the following option needs to be added: $OIFS_RUNCMD --allow-run-as-root $MASTER -e $EXPID
With the command 'exit' the container is removed and all created or changed files in the container are lost. The next section will show how results can be retained and OpenIFS experiments can be run using a container.
Docker Image with User Account
Running anDockerfiles with the note 'user' instead of 'root' in their filename contain an additional build step wherein a user account is created in the Docker image. In this case, once the image has been loaded as a container, we have no longer any root privileges as we have taken on the identity 'oifs_user'. Our default location is the home directory /home/oifs_user. We only have access to this home directory in the container, however as the model binaries were moved to /usr/local/bin they can be accessed. The model sources are hidden and a copy of the t21test directory is available in the home directory.
Running and Experiment in a Container
In this section we describe a method how the containerised version of OpenIFS can be used to run a case study on the user's workstation. Due to the temporary nature of containers all model results that were created in an experiment need to be stored outside the container. One possible method is to mount an external experiment directory inside the container. Data written to the mounted directory will be retained once the container is removed.
An experiment directory has been created and shall be located at /scratch/rd/user/exp/. This directory needs to contain all the necessary experiment data or symbolic links to further directories. This experiment directory is mounted to the container when it is invoked:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
A linked duplicate of the experiment directory can now be found inside the container in sub-directory /exp with read and write permissions. If a Dockerfile with user account is used then the experiment directory needs to be mounted within the user home directory of the container.
For a successful mounting of the directory it is necessary that the external experiment directory and all the files and or sub-directories therein have full read-write-executable access: chmod -R 777 /scratch/rd/user/exp
All files written by the model into the mounted directory will be owned by the container user. From outside the container the file owner will be different.
Crib Sheet: Important Docker commands
Start the Docker deamon on your machine (ECMWF):
| Panel | ||||
|---|---|---|---|---|
| ||||
|
which is actually: sudo /usr/bin/systemctl status docker
Which images are on my machine:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
Which containers are running:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
Build docker image:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
At ECMWF: docker build -t oifs --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy" .
Run docker images in container:
| Panel | ||||
|---|---|---|---|---|
| ||||
|
Use Harbor online container registry:
Do this first: docker login eccr.ecmwf.int
The build command below makes an image that can be pushed to harbor: docker build -t eccr.ecmwf.int/openifs/oifs:0.0.1 -f --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy"
Then push it to harbor, manually specifying version number. Careful: Existing version numbers are overwritten! docker push eccr.ecmwf.int/openifs/oifs:0.0.1
Pull image from repository into memory: docker pull eccr.ecmwf.int/openifs/oifs:0.0.1