You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 28 Next »

In this article we explain how to use OpenIFS in a container environment. At present this has been tested in two different ways using Docker on a Linux workstation:

  1. The user works interactively from inside the container and the external experiment directory is mounted as a sub-directory inside the container environment. Depending on the set up the user can either have access to the entire OpenIFS installation inside the container or the user may be prevented from accessing the source code. 
     
  2. The user only works from the experiment directory, and instead of executing the model binary program the OpenIFS run script starts up a container environment wherein the experiment runs in isolation. Immediately after the experiment has completed the container is removed. The user has no access to any part of the model installation. 

Motivation

Setting up the computing environment (the libraries, directory structure, etc) required by OpenIFS can present a challenge when it is necessary to run the model on a different hardware infrastructure, for instance during workshops and training events. It is time consuming to install and compile the model and all of its required software packages. Also, the libraries that are available on the local system may not be compatible with the model requirements.

Many of these issues can be avoided by running a containerised version of the model which is a self-sufficient code package that can be used in a consistent way on different hardware platforms. The computational overhead (the "costs") of the container environment itself is often outweighed by performance increases due to the local availability and the instant access to all required libraries and data within the container.

We have used the Docker platform to produce a container image for OpenIFS. This requires the design of a "Dockerfile" which describes the build process for the model code and all its dependent libraries, and which results in a binary Docker image. This image can be uploaded onto other computers that make use of the Docker platform or which use other compatible software. A “container” is the running instance of the Docker image. 

Pre-compiled OpenIFS Docker images can be made available for download from container image repositories (e.g. Docker Hub or Harbor) and should be able to run on any computer that uses the Docker platform without the need to install and compile the model or any additional software. The OpenIFS Docker images should also work with other container software compatible with the Open Container Initiative (OCI) standards such as Singularity or Sarus.


Contents

Crib Sheet: Important Docker commands


Start the Docker deamon on your machine (ECMWF):

sudo systemctl start docker
sudo systemctl restart docker
sudo systemctl status docker

which is actually:    sudo /usr/bin/systemctl status docker


Which images are on my machine:

docker images
docker rmi oifs                       remove image oifs, might need -f option 
docker rmi $(docker images -qa)       removes all images 
docker save -o oifs_image.tar oifs    saves image oifs to a tar file 
docker load -i oifs_image.tar         loads saved docker image into memory


Which containers are running:

docker ps
docker ps -a             show all containers
docker rm 6skd897asd     removes container beginning with 6sk...
docker rm $(docker ps -qa)   removes all containers


Build docker image:

docker build -t <image name> .        uses file called Dockerfile 
docker build -t <image name> -f <docker file>

At ECMWF:    docker build -t oifs --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy" .


Run docker images in container:

docker run -it ubuntu        run interactively with tty output
docker run -it oifs          run image oifs interactively
docker run -v /scratch/rd/damk:/scratch:rw -it oifs                                    mount volume $SCRATCH inside container
docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY metview metview    allows Metview to open X Window from inside the container


Use Harbor online container registry:

Do this first:   docker login eccr.ecmwf.int

The build command below makes an image that can be pushed to harbor:   docker build -t eccr.ecmwf.int/openifs/oifs:0.0.1 -f --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy"

Then push it to harbor, manually specifying version number.  Careful: Existing version numbers are overwritten!     docker push eccr.ecmwf.int/openifs/oifs:0.0.1

Pull image from repository into memory:    docker pull eccr.ecmwf.int/openifs/oifs:0.0.1

Dockerfiles

The Dockerfile describe the build process for the Docker image. Examples for several of these files are provided in the OpenIFS git repository.

The naming convention for Dockerfiles is as such:

Dockerfile.oifs<MODELRELEASE>.<GITHASH>.<NOTE>.<ARCH>.<TYPE>

MODELRELEASE:   A string generated from the IFS cycle, release number and OpenIFS version , e.g.  40r1v2 for CY40R1 OpenIFS release v2.

GITHASH:   The first five characters of the OpenIFS repository git commit from which this image is built.

NOTE:   An optional comment string that describes features of this build.

ARCH:  The architecture for which this image is built, e.g. x86_64, amd64, i386 etc.

TYPE:   dev, test or bld.  bld (build) to be used only for production images, such as from existing OpenIFS releases. dev should be used for images created from development branch commits.

Example:    Dockerfile.oifs40r1v2.41537.user.x86_64.bld

  • No labels