Crib Sheet: Important Docker commands
Start the Docker deamon on your machine (ECMWF):
sudo systemctl start docker
sudo systemctl restart docker
sudo systemctl status docker
which is actually: sudo /usr/bin/systemctl status docker
Which images are on my machine:
docker images
docker rmi oifs remove image oifs, might need -f option
docker rmi $(docker images -qa) removes all images, might need -f option
docker save -o oifs_image.tar oifs saves image oifs to a tar file
docker load -i oifs_image.tar loads saved docker image into memory
Which containers are running:
docker ps
docker ps -a show all containers
docker rm 6skd897asd removes container beginning with 6sk...
docker rm $(docker ps -qa) removes all containers, might need -f option
Build docker image:
docker build -t <image name> . uses file called Dockerfile
docker build -t <image name> -f <docker file>
At ECMWF: docker build -t oifs --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy" .
Run docker images in container:
docker run -it ubuntu run interactively with tty output
docker run -it oifs run image oifs interactively
docker run -v /scratch/rd/damk:/scratch:rw -it oifs mount volume $SCRATCH inside container
docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY metview metview allows Metview to open X Window from inside the container
Use Harbor online container registry:
Do this first: docker login eccr.ecmwf.int
The build command below makes an image that can be pushed to harbor: docker build -t eccr.ecmwf.int/openifs/oifs:0.0.1 -f --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy"
Then push it to harbor, manually specifying version number. Careful: Existing version numbers are overwritten! docker push eccr.ecmwf.int/openifs/oifs:0.0.1
Pull image from repository into memory: docker pull eccr.ecmwf.int/openifs/oifs:0.0.1
Dockerfiles
The Dockerfile describe the build process for the Docker image. Examples for several of these files are provided in the OpenIFS git repository.
The naming convention for Dockerfiles is as such:
Dockerfile.oifs<MODELRELEASE>.<GITHASH>.<NOTE>.<ARCH>.<TYPE>
MODELRELEASE: A string generated from the IFS cycle, release number and OpenIFS version , e.g. 40r1v2 for CY40R1 OpenIFS release v2.
GITHASH: The first five characters of the OpenIFS repository git commit from which this image is built.
NOTE: An optional comment string that describes features of this build.
ARCH: The architecture for which this image is built, e.g. x86_64, amd64, i386 etc.
TYPE: dev, test or bld. bld (build) to be used only for production images, such as from existing OpenIFS releases. dev should be used for images created from development branch commits.
Example: Dockerfile.oifs40r1v2.41537.user.x86_64.bld
Example Build Process
The following describes the build process of a container image from the Dockerfile. You need the following files in your build directory:
lrwxrwxrwx 1 45 Dockerfile -> Dockerfile.oifs40r1v2.415374d.root.x86_64.bld
-rw-r----- 1 2.1K Dockerfile.oifs40r1v2.415374d.root.x86_64.bld
-rw-r----- 1 21M oifs40r1v2.415374d.tar.gz
The Dockerfile can be obtained from the git repository. As described through the naming convention it will generate an image of OpenIFS 40r1 version 2. The partial git hash relates the Dockerfile (and image) to a specific git commit in the OpenIFS repository (in this case it relates to 415374d which is tagged as model release v2
). The note 'root
' indicates that when the image is loaded as a container the user will have root privileges. This will allow us to explore the directory structure of the container image. For convenience it is recommended to create a symbolic link to the generic name Dockerfile
.
The tar archive oifs40r1v2.41537.tar.gz
is created from the model sources after they have been checked out from the git repository (again the partial commit is specified). The Dockerfile will expect the tar archive in the same directory and the file name is specified.
The following command builds the image oifs40r1v2.415374d.root. The generic command is docker build -t <image_name>
however at ECMWF four variables need to be set for network proxies in order to access the internet from within the container.
docker build -t oifs40r1v2.415374d.root --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy" .
This starts the generation process of the image which contains the minimum of software that is required to run OpenIFS. The image is based on a Ubuntu Linux LTS version and in a first step the necessary developer tools are installed (e.g. GNU compiler, MPI and maths libraries). Afterwards the ecCodes library is obtained via download from the web and compiled with minimum settings. Then the OpenIFS sources are added from the tar archive, required environment variables are set and the model binaries are compiled. During a last step various file permissions are set and the model executable is moved to a globally accessible location.
At the end of the build process the successful image creation is shown as: Successfully tagged oifs40r1v2.415374d.root:latest
Dockerfiles with the note 'user
' instead of 'root
' in their filename contain an additional step wherein a user account is created in the image. In this case, once the image has been loaded as a container, the user has only access to their own home directory in the container and only the model binaries can be accessed. The model sources are hidden from the user.
Now we can verify the that the image is available and load it into a container:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
oifs40r1v2.415374d.root latest 982f6e82bb93 39 minutes ago 873MB
ubuntu latest 72300a873c2c 13 days ago 64.2MB
$ docker run -it oifs40r1v2.415374d.root
root@38b1649e05b9:/#
Our command line prompt has changed as we are now 'root
' inside the container. With the command 'exit
' the container is removed and all file changes in the container are lost. A file listing shows the directory structure inside the container.
root@38b1649e05b9:/# ls -F
bin/ dev/ home/ lib64/ mnt/ opt/ root/ sbin/ sys/ usr/
boot/ etc/ lib/ media/ oifs/ proc/ run/ srv/ tmp/ var/
The OpenIFS model is installed in /oifs
. The ecCodes library is found in its default destination under /usr/local/lib
.
In order to run the acceptance test the file /oifs/t21test/job
needs editing:
EXPID=epc8
MASTER=/usr/local/bin/master.exe
When setting GRIB_SAMPLES_PATH
replace grib_api
with eccodes
.
In order to run the executable with mpirun as root the following option needs to be added: $OIFS_RUNCMD --allow-run-as-root $MASTER -e $EXPID